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EXECUTIVE SUMMARY 



The U.S. Department of Agriculture (USDA) sponsored the NSLP Application/Verification 
Pilot Projects to test ways to improve the process for certifying students for free or reduced-price 
meals. This report presents findings on the impacts of two alternatives to the current application- 
based certification process — Up-Front Documentation and Graduated Verification — that were 
tested in 12 public school districts over a three-year period. 



Background 

Millions of U.S. children participate in the National School Lunch Program each day, 
receiving free or reduced-price lunches that make an important contribution to their overall 
nutrition. But concern has mounted that many of the children approved as eligible for free or 
reduced-price meals may in fact be ineligible because their family income is too high. Under the 
existing eligibility process, families are required to state their income on the application for 
benefits but do not need to submit any additional documentation. Districts select a small sample 
of applications for income verification, which is done later in the year. 

To address the question of whether the eligibility process could be made more accurate, the 
U.S. Department of Agriculture sponsored pilot projects testing two new approaches to certifying 
eligibility: (1) Up-Front Documentation, and (2) Graduated Verification. 

Districts using Up-Front Documentation required families to document their income or 
receipt of public assistance at the time they submitted their application for free or reduced-price 
lunches. Districts then used this documentation to make an eligibility determination, but did not 
verify any approved applications later in the school year. 

Districts using Graduated Verification allowed families to use the standard application 
process, which does not require income documentation, but changed key aspects of the usual 
verification process. After verifying a small sample of approved applications, these districts 
verified additional applications if 25 percent or more of the applications in the initial test resulted 
in benefit reduction or termination. 



Study Design and Methodology 

The study used a comparison design to select additional districts not participating in the 
three-year pilots but with similar economic characteristics and geographic locations. Researchers 
then compared the two types of districts to estimate impacts on the accuracy of the certification 
process, as well as to what degree it deterred ineligible families or discouraged eligible families 
from applying. Data for the study came from telephone and in-person interviews with about 
3,000 households with children enrolled in the study districts in fall 2002, and from 
administrative records provided by the schools. 
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Key Findings 



• Deterrence of Ineligible Families: Neither Up-Front Documentation nor Graduated 
Verification resulted in observable deterrence of erroneous certifications. The rates 
of erroneous certification among ineligible students were less than 5 percent in Up- 
Front Documentation comparison districts and less than 10 percent in Graduated 
Verification comparison districts. Neither Up-Front Documentation nor Graduated 
Verification had a statistically significant negative effect on the rate of erroneous 
certifications. In other words, neither pilot had a statistically significant deterrent 
effect. 

• Barriers for Eligible Families: Both sets of pilot procedures caused barriers among 
some eligible students. Rates of certification among each group of eligible students 
examined were lower in pilot districts than in comparisons districts. Some of these 
differences were statistically significant, indicating that Up-Front Documentation and 
Graduated Verification led to increased barriers among eligible students. 

• Accuracy Among Certified Students: Compared to current procedures, neither set of 
pilot procedures changed certification accuracy at a level that could be detected in 
the study. Overall, about 18 percent of students certified for free meals were ineligible 
for the benefits they were receiving. However, the estimated impacts of Up-Front 
Documentation and Graduated Verification on certification accuracy were small and 
not statistically significant. 
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SUMMARY OF FINDINGS 



The National School Lunch Program (NSLP) and School Breakfast Program (SBP) serve 
about 3.9 billion free and reduced-price meals to children annually. The food consumed at these 
meals makes up an important component of these children’s overall nutritional intake. In recent 
years, concerns have grown about the integrity of the program’s system for establishing 
eligibility for free or reduced-price meals. Several data sources suggest that the number of 
children approved for free or reduced-price meals from families with incomes too high to qualify 
for the benefits they receive is large and perhaps growing. 

The U.S. Department of Agriculture (USDA) sponsored the NSLP Application/Verification 
Pilot Projects to test ways to improve the process for certifying students for free or reduced-price 
meals. This report presents findings on the impacts of two alternatives to the current application- 
based certification process — Up-Front Documentation and Graduated Verification — that were 
tested in 12 public school districts. It examines the impacts of pilot procedures on three key sets 
of measures of program performance: 



1 . The rate of certification among ineligible households 

2. The rate of certification among eligible households 

3. Certification accuracy — the proportion of certifications that are correct 

In launching the pilots, USDA was seeking to identify changes to the certification process 
that would deter certification among ineligible households without causing barriers to 
certification among eligible households. The end goal was an overall improvement in 
certification accuracy as compared to the accuracy achieved with current certification 
procedures. Each pilot procedure was evaluated in terms of these measures. 



Current Certification Procedures 

Under federal guidelines, children living in families with incomes of 130 percent or less of 
the federal poverty level qualify for free meals, while those in families with income of between 
130 and 185 percent of poverty qualify for reduced-price meals. In addition, children receiving 
food stamp. Temporary Assistance for Needy Families (TANF), or Food Distribution Program 
on Indian Reservations (FDPIR) benefits are categorically eligible for free meals. For every 
NSFP meal served, FNS provides a reimbursement in cash and commodities whose amount 
depends on the child’s meal price status. For school year 2002-2003, the reimbursement rates 
were $2.34 for each free meal, $1.94 for each reduced price meal, and $0.36 for each paid meal. 

Most children who become certified for free meals do so because their family submits an 
application on which they report their income and household size or food stamp/TANF/FDPIR 
case number. Under standard federal rules, no documentation of applicants’ income or benefit 
receipt is required at the time of application. In the verification process, however, districts must 
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select a small sample of applications that have already been approved and collect income or 
benefit documentation from approved families in order to verify their eligibility for free or 
reduced-price meals. 



Pilot Certification Procedures 

Nine districts included in the evaluation implemented Up-Front Documentation. Under this 
pilot, districts required all applicants for free or reduced-price meals to provide documentation, 
either of their income or receipt of public assistance, with the application. Benefits were not 
granted if the application did not include the required documentation. After the applications and 
documentation were reviewed and approved, the districts were not required to conduct any 
subsequent verifications. 

Three districts included in the evaluation implemented Graduated Verification, under which 
the standard verification process was enhanced. Districts using Graduated Verification first 
conducted the standard verification process by verifying a small sample of approved 
applications. Unlike the standard process, however, these districts conducted up to two 
additional rounds of verification. If at least 25 percent of the initially verified applications had 
their benefits reduced or terminated as a result of the verification process, districts were required 
to verify an additional 50 percent of remaining applications. Similarly, if 25 percent of these 
second-round applications resulted in benefit reduction/termination, districts were required to 
verify all remaining applications. The parents of students whose benefits were 
reduced/terminated as a result of verification were required to submit documentation with their 
application if they applied for benefits in the following year. 



Design of the Impact Evaluation 

To estimate the impacts of these procedures, the pilot evaluation compared average 
outcomes among a sample of households in pilot districts with the outcomes of a sample of 
households in comparison districts. The comparison districts were selected to be similar to the 
pilot districts, except that they did not use the pilot procedures. 

Approximately 3,000 households with children enrolled in the 12 pilot and 12 comparison 
districts were interviewed in this study. The sample included approximately 1,300 households 
approved for free or reduced-price meals and approximately 1,700 households whose children 
were not approved for either free or reduced-price meals in October 2002. About two-thirds of 
the sample were in Up-Front Documentation pilot and comparison districts, and one-third was in 
the Graduated Verification pilot and comparison districts. 

The key data obtained in the survey were used to calculate an independent estimate of 
students’ eligibility for free or reduced-price meal benefits. The survey provided a consistent 
method of measuring household composition and household income by person and by source. 
The study acquired data from the school district on the actual approved meal price status of each 
sampled student. 




Changes in Key Administrative Measures Following Implementation of the Pilots 

USDA released a descriptive analysis of the first year of pilot project operations in 2002 
(U.S. Department of Agriculture 2002). USDA’s descriptive analysis of administrative data 
from the first year of pilot implementation showed that rates of certification for free and reduced- 
price meals declined following implementation of Up-Front Documentation and Graduated 
Verification, though USDA’s analysis did not determine whether the decline was driven by 
fewer ineligible students or fewer eligible students becoming certified. The analysis also showed 
that participation in the free and reduced-price categories declined following implementation of 
the pilots, while participation at the full-price level increased. This resulted in modest reductions 
in overall participation rates. 

In contrast to the administrative data used in the first- year analysis, the evaluation survey 
provides information on the underlying eligibility status of the households in the pilot and 
comparison districts during the third year of pilot implementation. This data can be used to 
assess the cumulative effects of using the pilot procedures for three years as compared to 
standard current procedures on the following: 



• Deterrence (lower certification rates among ineligible households) 

• Barriers (lower certification rates among eligible households) 

• Certification Accuracy (the proportion of certifications that are correct) 



What Were the Characteristics of Pilot and Comparison District Students? 

We examined the characteristics of the pilot districts and their populations and compared 
them with the nation as a whole to determine whether and how the districts that volunteered to 
test the pilot procedures might differ from the nation as a whole. Districts that implemented Up- 
Front Documentation were relatively well-off in terms of socioeconomic status. In the pre-pilot 
period, less than one-quarter of students were certified for free or reduced-price meals, compared 
with one-third of students nationally. The percentage of school-age children with incomes below 
100 percent of the federal poverty level in these districts was about 9 percent, compared with 15 
percent in the typical district nationally in 1999. The racial/ethnic distribution of the pilot 
districts also differed from that of the average district nationally — nearly 9 of 10 pilot district 
students were white, compared with 8 of 10 nationally. 

The number of students attending the average Up-Front Documentation district exceeded the 
number in the average public school district nationally. However, no very large districts 
implemented Up-Front Documentation. The largest of these pilot districts enrolled about 20,000 
students (although only one-third of its schools participated in the pilot). Although less than 2 
percent of districts nationally enroll more than 25,000 students, about one-third of all public 
school students nationally are enrolled in these very large districts. Furthermore, no Up-Front 
Documentation pilot districts were located in the central cities of metropolitan areas, though 
some were located in smaller cities adjacent to these central cities. 
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Graduated Verification pilot districts were more disadvantaged than the average public 
school district nationally. In 1999, 42 percent of students in these pilot districts were certified 
for free or reduced-price meals, and 22 percent of school-age children in them had incomes 
below 100 percent of the federal poverty level. Graduated Verification pilot districts had larger 
minority populations — 18 percent of students were black and 14 percent were Hispanic. 

We also compared the characteristics of students in the pilot and comparison samples in 
order to assess the extent to which our comparison district selection procedure produced samples 
of students with similar characteristics. A high degree of similarity is desirable because it 
supports using the experiences of the comparison group as a benchmark for what the experiences 
of the pilot group would have been if the pilot procedures had not been used. 

With a few exceptions, characteristics of students in the Up-Front Documentation pilot 
district sample were similar to those of students in the comparison district sample. However, the 
parents of pilot district students tended to be better educated than those of students in comparison 
districts, and pilot district households were also more likely to have incomes above 400 percent 
of the federal poverty level. Importantly, no statistically significant pilot-comparison differences 
in students’ pre-pilot certification rates or racial/ethnic distributions were found. Nor were there 
differences between pilot and comparison districts in the proportion of students whose low 
incomes make them eligible for free or reduced-price meal benefits. 

Similarly, few significant differences existed between the characteristics of students selected 
for the sample in Graduated Verification pilot districts versus the comparison districts. One of 
the few such differences was that pilot district sample members were somewhat less likely to be 
black than those in comparison districts. Again, the pre -pilot certification rates of the two groups 
were not statistically different from one another. 

We concluded that the pilot and comparison districts were reasonably well matched, and that 
they provide a reasonable basis for estimating demonstration impacts. However, the analysis 
underscored the importance of controlling statistically for individual differences in the 
characteristics of students and their families as we estimated demonstration impacts. 



Findings on the Impacts of Up-Front Documentation 

The evaluation’s impact analysis addressed the questions of how the pilot procedures 
affected deterrence and barriers, as well as two broader summary measures — program accuracy 
and targeting efficiency. To estimate impacts, we compared mean outcomes among students in 
pilot and comparison districts after controlling for the characteristics of these students’ 
households in a regression framework. The findings of the analysis of impacts of Up-Front 
Documentation are summarized below. 

In Up-Front Documentation pilot and comparison districts, well under 5 percent of all 
ineligible students were erroneously certified. Erroneously certified students were defined as 
those who were ineligible for free or reduced-price meals on the basis of their income as 
measured by the survey (in October and November for most students) but who were certified for 
free or reduced-price meals (as measured by district records provided to us in fall 2002). 
However, the number of erroneously certified students as a percentage of all certified students is 
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larger — for example, 18 percent of all students certified for free meals in comparison districts 
were ineligible for these benefits. 

Up-Front Documentation did not result in measurable deterrence of erroneous 
certifications. We defined deterrence as the difference between pilot and comparison districts in 
the rate of erroneous certification (certification of ineligible students). When students in 
households with income greater than 185 percent of poverty are considered, the proportion of 
ineligible students who were certified for free or reduced-price meals was the same in pilot and 
comparison districts after we controlled for household characteristics. On the other hand, when 
we focus on students above 130 percent of poverty — holding constant household 
characteristics — 4. 1 percent of students ineligible for free meals in comparison districts (that is, 
with incomes above 130 percent of poverty) were certified for these benefits; the percentage of 
ineligible students certified for free meals was lower in pilot districts, at 3.3 percent. This 
difference, while moderately large in percentage terms, was not statistically significant. 

Many students eligible for benefits were not certified. In both Up-Front Documentation 
pilot and comparison districts, substantial proportions of children eligible for free or reduced- 
price meals were not certified for these benefits. In comparison districts, for example, between 
half and two-thirds of all students eligible for free or reduced-price meals (excluding those 
directly certified and depending on the specific group considered) were certified, implying that 
one-third to one-half were not certified. 



Deterrence: Rates of Certification Among Ineligible 
Students (Up-Front Documentation) 



Percentage of students in 
households >185 FPL who 
were certified for free or 
reduced-price meals 

Percentage of students in 
households >130 FPL who 
were certified for free 
meals 

0 2 4 6 8 10 

Regression-Adjusted Percentages 

Note: Neither of the pilot-comparison differences shown is statistically significant at the .05 level. 
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The Up-Front Documentation Pilots caused barriers to certification among some eligible 
students. Up-Front Documentation was estimated to lead to a statistically significant reduction 
in the percentage of eligible students certified for free or reduced-price meals. Holding constant 
household characteristics, 42 percent of eligible students in pilot districts were certified for free 
or reduced-price meals, compared with 51 percent of those in comparison districts. Including 
directly certified students increased these percentages, but the percentages were in the same 
broad range and the pilot-comparison difference remained statistically significant. 



Barriers: Rates of Certification Among Non-Directly 
Certified Eligible Students (Up-Front Documentation) 







□ Pilot districts 


Percentage of students in 
households < = 185 FPL who 




42.0 


■ Comparison districts 



were certified for free or 
reduced-price meais 



51 .1* 



Percentage of students in 
househoids < = 130 FPL who 
were certified for free or 
reduced price meais 



53.1 




62.4 



* 



0 20 40 60 80 

Regression-Adjusted Percentage of Eligible Students Certified 



Note: ‘indicates the piiot-comparison differences is statisticaiiy significant at the .05 ievei. 



Compared to current procedures, Up-Front Documentation did not change accuracy or 
targeting efficiency at a level that was statistically significant. The estimated impact of Up- 
Front Documentation on the proportion of certified students eligible for the benefits they were 
receiving depended on whether we were looking at free meal certification or free and reduced- 
price meal certification. While the pilot was estimated to have a positive impact on the accuracy 
rate for free meals, its estimated impact on the accuracy rate for free or reduced-price meals was 
negative. Neither estimate was statistically significant. The estimated impact of Up-Front 
Documentation on targeting efficiency was negative but small and not statistically significant 
(data not shown). Holding constant household characteristics, for example, the proportion of all 
non-directly certified students with a free/reduced-price meal certification status consistent with 
their eligibility status was 79 percent in pilot districts and 81 percent in comparison districts. 
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Accuracy Rates Among Certified Students — Excludes 
Directly Certified Students (Up-Front Documentation) 
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Regression-Adjusted Percentages 



Note: Neither of the pilot-comparison differences is statistically significant at the .05 level. 



Findings on the Impacts of Graduated Verification 

In Graduated Verification comparison districts, less than 10 percent of ineligible students 
were erroneously certified. Among students ineligible for free or reduced-price meals on the 
basis of their income, just under 10 percent were certified for free or reduced-price meals, while 
9 percent of students ineligible for free meals were certified for free meals. Again, the number 
of erroneously certified ineligible children is larger as a percentage of all certified students — for 
example, 18 percent of students certified for free meals (excluding directly certified students) 
were ineligible for these benefits. 

The deterrent effects of Graduated Verification were not statistically significant. Holding 
constant household characteristics, 8.6 percent of students ineligible for free or reduced-price 
meals in comparison districts were certified, while the certification rate among this group in pilot 
districts was only 6.1 percent. However, this difference was not statistically significant at the 
0.05 level. Among students ineligible for free meals, 8.0 percent were certified for free meals in 
both pilot and comparison districts. 

The Graduated Verification pilots caused barriers to certification among some eligible 
students. Among students eligible for free meals and not directly certified, the percentage 
certified for free or reduced-price meals was 64 percent in pilot sites and 80 percent in 
comparison sites. This 16 percentage point difference was statistically significant at the .01 
level. Among those eligible for free and reduced-price meals who were not directly certified, the 
certification rate was 62 percent in pilot districts and 71 percent in comparison districts. This 9 
percentage point difference was not statistically significant at the .05 level. 
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Deterrence: Certification Rates Among Ineligible Students 
(Graduated Verification) 

Percentage of all students 
In households >185 FPL 
who were certified for free 
or reduced-price meals 

Percentage of students In 
households >130 FPL who 
were certified for free 
meals 
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Regression-Adjusted Percentages 

Note: Neither of the pilot-comparison differences shown is statistically significant at the .05 level. 



Barriers: Certification Rates Among Non-Directiy 
Certified Eiigibie Students (Graduated Verification) 

Percentage of students in 
households <=185 percent 
FPL who were certified for 
free or reduced-price meals 



Percentage of students in 
households <=130 percent 
FPL who were certified for 
free or reduced price meals 
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Regression-Adjusted Percentage of Eligible Students Certified 



Note: ''indicates the pilot-comparison difference is statistically significant at the .01 level. 
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Graduated Verification did not change accuracy or targeting efficiency at a statistically 
significant level. The estimated impaet of Graduated Verifieation on aeouraey was negative but 
not statistically significant among students certified for free meals, and positive but not 
statistically significant among students certified for free or reduced-price meals. The estimated 
impact of Graduated Verification on targeting efficiency was small, negative, and not statistically 
significant. 



Accuracy Rates Among Certified Students — Excludes 
Directly Certified Students (Graduated Verification) 



Percentage certified for 
free or reduced-price 
meais who had income 
< = 185 FPL 



Percentage certified for 
free meais who had 
income < = 130 FPL 



88.9 




70.5 




74.3 



□ Pilot districts 
■ Comparison districts 
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Regression-Adjusted Percentages 



Note: Neither of the pilot-comparison differences is statistically significant at the .05 level. 
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I. INTRODUCTION 



The National School Lunch Program (NSLP) and the School Breakfast Program (SBP) are 
important elements of the U.S. system for safeguarding the health and well-being of its school- 
age children. In fiscal year 2002, the programs served more than 3.9 billion free or reduced- 
price school meals to students (U.S. Department of Agriculture, Food and Nutrition Service 
Web Site, August 2003). They provided an important component of the overall nutritional intake 
of many of these children. 

Efficient, accurate administration of the program is essential for ensuring effective targeting 
of benefits and high levels of public confidence in its operations.^ In recent years, however, 
concerns have been raised about the integrity of the program’s process for establishing eligibility 
for its benefits. In response, the U.S. Department of Agriculture (USDA) has sought the 
voluntary participation of School Food Authorities (SFAs) to test ways of improving the process 
for certifying students to receive free and reduced-price meals. This report presents the results of 
an evaluation of two of the approaches tested in the NSFP Application/Verification Pilot 
Projects: Up-Front Documentation and Graduated Verification. The evaluation is based on data 
collected during school year 2002-2003, the third year of pilot operations. 

The rest of this introduction describes the policy context and problem, the design of the 
pilots, and the policy questions that the evaluation will address. 



*For brevity, we refer in this report to eligibility and certification for the NSFP. Students 
approved for free or reduced-price lunches also qualify automatically for free or reduced-price 
school breakfasts. 
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A. IMPORTANCE OF PROGRAM INTEGRITY 



The NSLP was enacted with the passage of the National School Lunch Act of 1946 to 
“safeguard the health and well-being of the Nation’s children and to encourage the domestic 
consumption of nutritious agricultural commodities and other foods.” In 1975, Congress 
expanded the federal role in providing students with access to nutritious food by authorizing the 
creation of a permanent school breakfast program: the SBP. The NSLP and SBP provide federal 
financial assistance and commodities to schools serving meals that meet specified nutritional 
standards. Although USDA subsidizes (with cash reimbursements and commodities) all school 
lunches and breakfasts, the subsidies are largest for children approved for free or reduced-price 
meals. Children in families whose incomes are 130 percent or less of the federal poverty 
guidelines or who are members of families receiving benefits of the Food Stamp Program, 
Temporary Assistance for Needy Families, or the Food Distribution Program on Indian 
Reservations (FS/TANF/FDPIR) qualify for free meals; those in families with incomes between 
130 and 185 percent of the guidelines qualify for reduced-price meals. For every NSLP meal 
served, FNS provides a reimbursement in cash and commodities whose amount depends on the 
child’s meal price status. For school year 2002-2003, the reimbursement rates were $2.34 for 
each free meal, $1.94 for each reduced-price meal, and $0.36 for each paid meal. 

I. Reasons for Policy Concern 

Like all programs that use means tests to direct benefits to low-income households, the 
school nutrition program must balance competing objectives: (1) ensuring that approved 

children are income eligible, (2) maintaining ease of access for eligible children, and (3) keeping 
the costs and burdens of administering eligibility determination reasonable both for SFAs and for 
families. Meeting the first objective can sometimes increase administrative costs and make it 
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more difficult for eligible children to participate. Making access simpler or streamlining 
administration might result in more benefits going to people who do not qualify for them. 

Accurate certification of children for free and reduced-price school meals is important for 
two reasons. First, continued public support for the NSLP requires an eligibility determination 
process that keeps erroneous payments — that is, reimbursements for free or reduced-price meals 
obtained by children from households that are not income eligible — to a minimum. The federal 
cost of the NSLP and SBP was approximately $7.6 billion in fiscal year 2000. Second, many 
federal and state programs designed to provide additional resources to districts with large 
numbers of at-risk students now allocate funding on the basis of the number of children approved 
for free and reduced-price school meals. For example, states and districts frequently allocate 
federal funding under Title I of the Elementary and Secondary Education Act of 1965 (20 U.S.C. 
2701 et seq.) to individual schools according, in part, to the percentage of students approved for 
free and reduced-price meals. States and districts also allocate other state-funded education 
programs on the basis of these school-level measures. 

While it is important to minimize the number of ineligible children certified for free and 
reduced-price meals, policymakers are concerned that eligibility determination procedures do not 
create significant barriers to the certification and participation of eligible children. Indeed, such 
barriers defeat a key objective of means-tested programs: to make sure these public programs 
are accessible to all who are eligible. 

2. Concerns About Inaccuracy 

Several studies over the past 25 years have found that a substantial number of children from 
ineligible households were certified for free or reduced-price meals. These findings have 
contributed to the interest in reducing certification errors. A study sponsored by USDA’s Eood 
and Nutrition Service (ENS) (U.S. Department of Agriculture 1990) examined the outcomes of 
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verification reported by a representative sample of SFAs and in-home interviews with a 
representative sample of households that did not respond to the SFA’s verification request. Data 
presented in this study suggest that about 15 percent of children certified as of December 1986 
were in households not eligible for the benefits they had been approved for in early fall 1986. 
In an audit covering school years 1994-1995 and 1995-1996 for a representative sample of 
Illinois SFAs, USDA’s Office of Inspector General (1997) estimated that 19 percent of all 
verified children had their benefits reduced or terminated. 

FNS also was concerned that the number of children approved for free meals nationally 
exceeded the estimated number of children in families with annual income less than 130 percent 
of the federal poverty level. This evidence and other program oversight activity suggested that a 
substantial number of households misreport eligibility information in order to be approved for 
free and reduced-price school meals. ^ 



B. THE PILOT DEMONSTRATIONS AND KEY POLICY QUESTIONS 

To address concerns about misreporting and approval of ineligible children, in January 
2000, FNS invited state agencies and SFAs to test one of four specific new strategies (or propose 
their own alternative strategy). Two of these four strategies, Up-Front Documentation and 



From U.S. Department of Agriculture 1990, Exhibit 5.1, 10.9 percent of cases verified in 
December 1986 in districts using random sampling responded and had their benefits reduced or 
terminated, and 12.9 percent had their benefits terminated because they did not respond to the 
SFA’s request for verification. From Exhibit 5.4, in-home audits conducted in May 1987 found 
that one-third of nonresponders whose benefits were terminated were not eligible for the benefits 
they were approved for in early fall 1986. Multiplying this percentage ineligible by the 12.9 
percent who were nonresponders and adding the product to 10.9 gives an estimate of 15.2 not 
eligible for the benefit they were receiving. It should be noted that this very likely overstates the 
percentage not eligible for their benefits in December 1986 because some of the cases found 
ineligible as of the May 1987 interview may have been eligible in December 1986 but 
experienced a change in income between December and May. 

^Federal Register, vol. 65, no. 14, January 21, 2000, p. 3410. 
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Graduated Verification, are examined in the evaluation reported herein. Up-Front 
Documentation strengthens the application process by requiring documentation at the time of 
application. Graduated Verification strengthens the verification process through graduated 
increases in the number of cases verified if the percentage of cases whose benefits are reduced or 
terminated as a result of verification exceeds specific thresholds. As background for 
understanding the approaches to improving the application and verification process and their 
potential effects, we briefly describe the current rules and procedures. 

1. Background on Current Application and Verification Procedures 

Responsibility for administering the NSLP and SBP at the district level rests with the SFA, 
which provides meals that meet nutritional requirements and, for purposes of claiming 
reimbursement from USDA, keeps counts of meals served free, at reduced price, and at full 
price. The SFA also is responsible for (1) notifying fa mi lies that free and reduced-price meals 
are available to children from families that meet certain income-for-family-size standards; 
(2) distributing blank application forms and instructions, receiving completed applications, and 
informing families that they must report to the SFA any increase in their income of $50 per 
month or greater; (3) reviewing the information on forms to establish which students are 
approved (certified) and which applying students are not eligible (and, therefore, are not 
certified); and (4) notifying fa mi lies of their children’s certification status. 

The current verification process requires that each SFA conduct verifications of selected 
applications by December 15 of each year."^ Districts initiate the process by sending the family a 

"^SFAs have the option of conducting verification with a random sample of applications, 
with & focused sample of applications, or with all applications. A random sample is supposed to 
be a simple random sample of all approved applications. If a random sample is used, it must 
include the lesser of 3,000 or 3 percent of applications. In a focused sample, verification efforts 
center on students in families that have incomes close to the upper cutoff of eligibility or that 
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request for documentation of their income or food stamp/TANF status. If the family provides 
documents that show that their income exceeds the eligibility limit, their benefits are reduced or 
terminated. If they fail to provide documentation, the SFA is required to terminate benefits. 

An important feature of the application process is that SFAs are permitted (but not required) 
to use “direct certification” for certifying approval of children from families that receive food 
stamps or cash assistance. Under direct certification, the food stamp or welfare agency identifies 
to the school district those children in families that receive these income supports; the school 
district can approve these children for free meals with no application. SFAs are not required to 
perform verification for students approved for free meals by direct certification. 

2. Key Features of the Demonstrations Being Evaluated 

A clear picture of how the pilot procedures changed existing processes is necessary to 
understand the evaluation findings and to assess how they might apply if they were implemented 
nationally. Figure I.l shows the processes and highlights the distinctive features of each pilot 
model. 

Nine SFAs in the evaluation tested Up-Front Documentation during school year 2000-2001 
through school year 2002-2003. Under Up-Front Documentation, SFAs required all applicants 
for free and reduced-price meals to provide, with the application, documentation either of their 
income or of their receipt of TANF or food stamp benefits. Benefits cannot be granted if the 
application does not include a complete application, including documentation. After the 
applications and documentation are reviewed and approved, the SFAs need not conduct any 



(continued) 

receive TANF or food stamps. Such a sample must include the lesser of 1 percent of all 
applications or 1,000 applications selected from families whose income is within $100 of the 
income limit for their family size, plus the lesser of 0.5 percent of all applications or 500 
applications selected from those receiving TANF or food stamps. 
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subsequent verifications. The requirement for Up-Front Documentation applies only to families 
who submit an application; no documentation is required for students approved through direct 
certification. 

Three districts participating in the evaluation tested Graduated Verification, under which 
application procedures are strengthened and, in certain circumstances, the verification process is 
enhanced. First, households who were applying for free or reduced-price meals and whose 
benefits had been terminated or reduced in the prior year as a result of SFA verification activities 
were required to provide documentation of their incomes or their categorical eligibility at the 
point of application. Second, in addition to conducting the standard verification of 3 percent of 
participating households, the SFA was required to conduct additional verifications as follows: 

• If 25 percent or more of the initially verified applications led to a termination or 
reduction of free or reduced-price meal benefits, the SFA had to verify an additional 
50 percent of the remaining applications. 

• If 25 percent or more of these second-round verifications resulted in terminations or 
reductions in benefits, the SFA had to verify all the remaining applications. 

In assessing how the pilots can be expected to affect the accuracy of the benefit 
determination process, it is useful to consider the possible sources of inaccuracy identified 
through the current verification process. These include: 

• Intentional misreporting of circumstances at the time of application 

• Inadvertent misreporting of circumstances at the time of application 

• SFA errors in processing information provided by the household about their 
circumstances 

• Unreported increase in household income of more than $50 per month or departure of 
a household member between application and the point at which verification is 
conducted 
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Requiring documentation could deter intentional misreporting, as households are less willing 
to understate the amount of income from a given source. However, it would not detect the 
situation in which a household reports and documents one income source but conceals another. 
Requiring documentation at application could reduce inadvertent misreporting if families do not 
understand the distinction between their net income and their gross income (the latter is used in 
meal price certification). 

Increasing the number of applications verified increases the likelihood of detecting 
inaccuracies due to changes in household income or composition. The pilot projects may also 
affect the application behavior of eligible families, the most likely potential effect being that 
some eligible families would be discouraged from applying. The likely effects on administrative 
error are uncertain. 

3. Administrative Data on Changes in Certification Following Implementation of the 

Demonstrations 

USDA published a report on the first-year experiences of SFAs implementing Up-Front 
Documentation and Graduated Verification (USDA 2002).^ The analysis compared 
administrative data on the number of students approved for free and reduced-price meals and the 
numbers of meals served before the pilots were implemented and during the first year of pilot 
operation. In Up-Front Documentation pilot districts, the numbers of students approved declined 
relative to averages in the two years prior to the demonstration. The number of free approvals 
declined by 20 percent, and the number of reduced-price approvals declined by 9 percent. The 

^The report also examined the experiences of seven SFAs that implemented pilot procedures 
whereby verifications were conducted for students approved by direct certification. On the basis 
of findings that rates of ineligibility were very low (6.6 percent of those verified had their 
benefits reduced or terminated), FNS decided not to include the Verify Direct Certification pilots 
in the present evaluation. 
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number of free meals served dee lined by a proportion similar to the ehange in free eertifieations, 
the number of reduced-priee meals was essentially unchanged, and the number of paid meals 
increased. 

In Graduated Verification districts, the analysis examined experience near the end of the 
first year of implementation in the four pilot districts. Just over one-third of children in these 
districts (36 percent) who were approved by application and selected for verification experienced 
a reduction or termination in their benefits. In the three districts, in which initial verification 
results triggered expanded verification, the number of free meals served in April and May 2001 
declined by 20 percent compared to a similar period in the prior two years, and the number of 
reduced-price meals declined by 8 percent, while total meals declined by just 1 percent. 

4. Research Objectives 

The pilot evaluation extends the analysis that was possible using administrative data for the 
pilot sites during the first year of operation. It examines the underlying eligibility status of 
households in the pilot districts and in similar comparison districts during the third year of pilot 
implementation. The evaluation data can be used to assess the cumulative effects of operating 
the pilot procedures on deterrence (lower certification rates among ineligible households), 
barriers (lower certification rates among eligible households), and certification accuracy (the 
proportion of certifications that are correct). 

The evaluation of the Up-Front Documentation and Graduate Verification pilot projects 
addressed the following six objectives and associated questions: 

1. Assess Deterrence and Barrier Effects of the Demonstration. Did the 
demonstrations reduce the likelihood that an ineligible household would apply for 
and have their child be approved for free or reduced-price school meals? Did it 
reduce the likelihood that an eligible student would be approved for free or reduced- 
price school meals? 
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2. Assess Impacts on the Accuracy and Targeting Efficiency of Certification. How 
did the application/verification pilot demonstrations affect the proportion of children 
who were approved for free or reduced-price meals (based on an application) that 
were eligible for each level of benefit? Did it affect the percentage of all students 
who were certified and eligible or not certified and ineligible? 

3. Assess the Effects of the Demonstration on NSLP Participation. How did the 
pilots affect participation in the NSLP, as measured by the numbers of free, reduced- 
price, and paid meals consumed relative to the number of students in the school? 

4. Assess the Administrative Burden of Demonstration Procedures. What costs do 
SFAs incur in administering the free and reduced-price meal certification process, 
and how does each of the application/verification processes being tested change 
administrative burden? 

5. Assess the Fidelity of Implementation. Did the pilot sites administer the 
demonstration procedures in accordance with the agreed-to protocols? What 
problems were encountered in implementation, and how were they resolved? 

6. Describe the Characteristics of Students. What are the characteristics of students 
approved for free meals, approved for reduced-price meals, and not approved for 
either in terms of racial/ethnic group, type of location, income group, and other 
characteristics of the household? How do the characteristics of students in the pilot 
districts compare with those in comparisons and with the country as a whole? 



This report presents the study findings on the impacts of the pilot projects on deterrence, 
barriers, accuracy, and targeting, and describes the characteristics of students in the pilot 
districts. Subsequent reports will present findings on the demonstrations’ effects on applications, 
participation, and implementation and administrative costs. 
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II. STUDY BACKGROUND 



This chapter provides background on the study. The first section briefly describes the study 
design, while the second describes the pilot sites in which the study was implemented. 
Additional details on these topics are contained in Volume II of this report. 

A. OVERVIEW OF STUDY DESIGN 

To estimate effects of the pilot demonstrations on deterrence, barriers, accuracy, and 
targeting efficiency, we used a comparison district design. This design requires comparing 
outcome measures for students potentially affected by the demonstrations with those of 
otherwise similar students who were not exposed to the demonstration rules. In particular, a 
single, carefully matched comparison SFA was recruited to serve as the benchmark for each pilot 
SFA. We selected representative samples of students and collected data in the same manner in 
the comparison SFAs as in the pilot SFAs. We calculated estimates of impacts on measures of 
barriers and deterrence by computing the average value of the key outcomes in each pilot and 
comparison district (after controlling for key student characteristics), computing the difference 
between the pilot and comparison district in each site pair, and then computing the mean of these 
differences. We derived estimates of the impacts on accuracy and targeting efficiency from 
these estimated impacts on barriers and deterrence. We present separate impact estimates for the 
Up-Front Documentation and Graduated Verification pilot projects. 

The remainder of this section provides key details of this approach to measuring the impacts 
of Up-Front Documentation and Graduated Verification. First, we describe the selection and 
recruitment of comparison districts. Next, we present the process by which we designed and 
selected the sample of students to be included in the analysis. The third section summarizes the 
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data collection process. Finally, we describe various aspects of our strategy for estimating the 
impacts of the pilot demonstrations. 

1. Selection and Recruitment of Comparison Districts 

The selection of comparison sites was guided by several criteria. It was especially important 
that each comparison be similar to its pilot district in having (1) similar NSLP application and 
eligibility determination procedures, as measured by the use of direct certification; (2) similar 
rates of free and reduced-price certification prior to the pilot; (3) similar district enrollment and 
grade span, and racial and ethnic composition; and (4) similar percentages of students from 
families with incomes below 130 percent of the federal poverty level. We also sought 
comparison districts that were geographically close to the pilots and were similar in terms of 
their urban/suburban or rural setting. It was also important that each choice be viewed as 
reasonable by knowledgeable local people. 

To develop our list of comparison district candidates, we followed a three-step process that 
started with all public school districts nationwide. First, we restricted the choice set for each 
pilot to include districts within the same state and districts serving the same grade span.^ 
Second, we created a quantitative index of similarity for each district that remained a candidate 
after the above restrictions were imposed. The index normalized (to remove the effects of scale 
for different measures included in the index), and then weighted, the various characteristics 
included in the index. The scale included six components: (1) size, (2) race/ethnicity, (3) 
poverty, (4) proximity, (5) level of certification in the NSLP, and (6) level of participation in the 

*We initially planned to restrict the candidate pool to school districts with similar 
urban/rural/suburban status as measured in the NCES Common Core data set. However, initial 
testing revealed that this restriction eliminated some plausible candidates that we felt should be 
included, so we relaxed this requirement. 
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NSLP. We experimented with different algorithms for weighting the various eomponents. Our 
preferred weighting gave 25 pereent of the weight each to the poverty, certification, and 

'•y 

participation components; 15 percent to proximity; and 5 percent each to size and race/ethnicity. 

For the third step in the selection process, we asked people who know the areas in which the 
pilot districts are located, and know the schools in those areas, to provide their assessment of 
whether each candidate district on our short list was a reasonable choice. That is, we asked 
whether each candidate would be viewed by a knowledgeable person as providing a valid 
benchmark for the pilot district. We also asked whether any of the candidates were obviously 
poor choices because of differences that were not apparent from the data available to select the 
short list. After completing this third step, we arrived at a decision on the top candidate (or, in 
some cases, the top two candidates), and then asked these districts to participate in the study. 

The process for asking districts to participate as comparison districts varied by state. In 
some states, the state coordinator of the NSLP (or someone from that person’s office) made the 
request on behalf of the study, while in others a person from the pilot site who knew a senior 
official at the candidate comparison site made the first contact. 

If a candidate district refused, we identified additional candidates and/or devised alternative 
ways to gain entry with the potential comparison district to have our request considered by 
someone in a position to make a decision on the district’s behalf. For half of the 12 pilot districts 
included in the evaluation, we successfully recruited the first candidate that we approached. For 
two pilot districts, we had to approach two comparison district candidates because the first 
turned us down, and for three pilot districts, we had to approach three candidates because the 
first two turned us down. For one of the pilot districts, we approached seven potential 

2 

The other weighting systems considered gave similar rankings. 
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comparison districts before finding a willing participant. Most districts that declined to 
participate did so because of concerns that the community would view the study interview as 
intrusive. Volume II provides additional details on these efforts. 



2. Sample Design and Sample Selection 

The study ultimately included (1) nine Up-Front Documentation pilot districts plus one 
comparison district for each pilot, and (2) three Graduated Verification pilot districts plus one 
comparison district for each of two pilots and two comparison districts for one pilot district. The 
next step was to select a sample of students. This sample was designed to be representative of all 
students who were not approved for free meals by direct certification in each pilot district and its 
comparison district and who were enrolled in the school district at the time of data collection. 
For purposes of efficiency, FNS decided to exclude from the survey sample all students who 
were directly certified. The sample was stratified by district and by meal price status. Within 
each pilot model group, approximately equal numbers of observations were allocated to each 
district and to each meal price status group within it. 

The sampling and data collection schedule was designed so as to meet the objective 
providing FNS with preliminary tabulations of the data by the end of February 2003. According 
to this schedule, the data collection occurred between October and December 2002, at about the 



The incomes of directly certified students are subject to careful assessment and 
documentation as a condition for receiving TANF or food stamp or FDPIR benefits. 
Furthermore, FNS found that by December of the school year 4.1 percent of students directly 
certified prior to the beginning of the school year remained enrolled and had their benefits 
reduced or terminated (Food and Nutrition Service, Office of Analysis, Nutrition, and 
Evaluation, 2002). This finding suggests that only about 4 percent of directly certified students 
remained enrolled and experienced an increase in income to above 130 percent of poverty. 
Therefore, including these students in a survey designed primarily to measure income relative to 
poverty would have provided less information than a larger number of interviews with the 
students who were not directly certified. 
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same time that the verification process takes place in most districts. To meet this schedule, we 
began obtaining the lists necessary for sampling right at the beginning of the school year. For 
most districts, we acquired four separate lists of (1) all students enrolled in the district, (2) all 
students directly certified, (3) all students approved for free meals, and (4) all students approved 
for reduced-price meals. Lists varied in their format and content, and as to whether they were 
hard copy or electronic. All full enrollment lists and all lists of students directly certified were 
for the current school year. Because of the accelerated sampling and data collection schedule, 
however, the lists of students approved for free or reduced-price meals used for sampling 
purposes in some districts were as they existed at the end of the 2001-2002 school year. 

Processing lists for sample selection entailed identifying and removing duplicate entries on 
each list, and matching each of the three meal price status lists with the student enrollment list. 
This process eliminated from the sampling frame any students who were directly certified, and 
created three sampling strata: (1) students approved for free meals on the basis of an application, 
(2) students approved for reduced-price meals, and (3) the balance (students paying full price). 

The initial samples included a total of 3,685 students: 1,178 in nine Up-Front 
Documentation pilot districts, 1,201 in nine Up-Front Documentation comparison districts, 637 
in three Graduated Verification pilot districts, and 669 in four Graduated Verification 
comparison districts.^ 



'^The details of this process varied across districts and are presented in Volume II. 

^Two neighboring districts served as the comparison for the Dilworth-Glyndon-Felton 
district in Minnesota: half the comparison sample was selected from each of the comparison 
districts. 
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3. Data Collection 



The key data for the analysis of impaets are measures of household ineome and number of 
persons in the household, which are needed to classify each sample household as having 
(1) income less than 130 percent of the federal poverty level (eligible for free meals), (2) income 
between 130 and 185 percent of the federal poverty level (eligible for reduced-price meals), or 
(3) income above 185 percent of the federal poverty level (not eligible for free or reduced-price 
meals). The study’s survey was conducted primarily to gather the household size and income 
information necessary to classify each member of the sample into the correct income eligibility 
category. The survey was also used (1) to gather information about experiences with the lunch 
program, (2) to obtain parents’ perceptions of the program, and (3) to collect data that relate to 
the characteristics of households that could be used to adjust statistically for differences in the 
average characteristics of students in the pilot sample and students in the comparison sample. 

An extensive literature on the accuracy of income reporting in surveys indicates that 
responses to questions asking “What was your household’s total income for last month?” are 
subject to considerable reporting error.^ Better estimates are obtained by asking whether each 
household member had income for each of several possible sources of income and, if so, what 
amounts each person received from each source. Having survey respondents obtain and refer to 
pay stubs or other written documentation also improves the accuracy of income reporting (Bogen 
et al. 1992). Income reporting errors in surveys tend to be greater proportionately among lower- 
income families, in part because the incomes of such families are often less regular and more 
variable. 



^Marquis et al. 1993 and 1994; and Moore et al. 2000. 
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In light of this collective experience (and given the study’s objectives), we used a two-step 
interview process designed to minimize the burden on families in responding to the survey and to 
obtain the necessary data efficiently, while allowing us to accurately classify each family’s 
income status. In the first step, we conducted a telephone interview with the parent or guardian 
of each student selected for the study sample. In the telephone interview, respondents were 
asked about experiences with and perceptions of the NSLP and about characteristics of their 
family. They were also asked one question about the number of persons in their household and a 
short series of questions about their total family income that allowed us to classify their family 
income as above or below 400 percent of the federal poverty level. 

If in the telephone interview the reported family income was above 400 percent of poverty, 
for analytical purposes we classified the family as having income above 185 percent of poverty. 
We chose 400 percent of poverty as the cutoff because we judged that very few respondents 
whose true income was less than 185 percent of the federal poverty level would report an income 
as high as 400 percent of poverty in response to the global question about total family income. 

If in the telephone interview the parent reported a family income of less than 400 percent of 
poverty, we attempted to conduct an in-depth in-home interview with the family. This interview 
asked a detailed set of questions about the persons living in the household and about their 
economic relationships. Then, for each adult who was part of the economic unit to which the 
sampled student belonged, we asked the respondent to identify sources of income and to provide 
details about each source, including the amount. We also asked respondents to retrieve 
documentation of amounts, such as pay stubs, and then to refer to these documents in giving the 
amounts of income by person and source. 

Survey data collection started on October 16, 2002, and was completed on January 6, 2003. 
Most interviews were conducted in November and December 2002. Respondents were asked to 
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report on household income in the month prior to the month in which the interview was 
conducted. This reference month was typically October (57 percent) or November (30 percent), 
but was September (13 percent) or December (less than 1 percent) for a small proportion of 
respondents. 

4. Estimating Impacts on Deterrence, Barriers, Accuracy, and Targeting Efficiency 

Our basic approach to estimating impacts was to compare the mean values of the outcome 
measures (as shown in Table IV. 2) in each pilot-comparison site pair, and then to compute the 
average of these pilot-comparison differences. This process was straightforward in the case of 
estimating impacts on deterrence and barriers. For example, to estimate effects on deterrence for 
students in families with income exceeding 185 percent of the federal poverty level in Up-Front 
Documentation districts, we computed the percentage of students in households with income 
exceeding 185 percent of poverty who are approved for free or reduced-price meals in each of 
the nine pilot and comparison districts, computed the difference between this measure in each 
pilot and comparison site pair, and calculated the mean of these nine differences. In calculating 
the differences between pilot and comparison site pairs, we controlled in a regression framework 
for exogenous student and household characteristics that potentially could influence the outcome. 
Each pilot-comparison site pair contributes equally to the overall impact estimate. Once the 
impacts on deterrence and barriers were estimated, we derived the impacts on accuracy and 
targeting efficiency from those previous estimates, as described below. 

a. Measuring Outcomes of Interest 

To estimate impacts on deterrence, barriers, accuracy, and targeting efficiency, we first had to 
construct variables that would operationalize these concepts. Each of these general outcomes 
depends in some way on individual students’ certification status — whether they are approved for 
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free or reduced-price meals — and household income — whether they are income eligible for free 
or reduced-price meals. Table II. 1 shows in tabular format various combinations of students’ 
certification status and income that are relevant for the study. The table is designed so that all 
enrolled students in a district can be assigned to one of the cells. The rows of the table show 
students’ certification status — whether students are directly certified, certified free by 
application, certified reduced-price, or not certified. The columns shows their household 
income — whether they are income-eligible for free meals (income no more than 130 percent of 
poverty), eligible for reduced-price meals (income between 131 and 185 percent of poverty), or 
not eligible for either (income above 185 percent of poverty).’ 

Variables measuring our outcomes of interest can be represented by counts of students in 
particular rows and columns of Table ILL To measure free/reduced-price deterrence, for 
example, we use the certification rate among students with incomes above 185 percent of 
poverty. This measure can be represented by the fraction (D-i-H)/P, since P represents the total 
number of students with incomes above 185 percent of poverty and (D-i-H) represents the number 
of these income-ineligible students who are certified for free or reduced-price meals. Other 
outcome variables can be defined similarly. 

Concerns about the level of inaccuracy in certification for free and reduced-price meals 
motivated the testing of pilot procedures to improve certification accuracy. The central questions 
surrounding accuracy addressed in the evaluation are: What percentage of students certified for 
free and reduced-price meals are ineligible for the benefits they receive? How did the 



Students’ eligibility for free meals also depends on whether they are receiving 
FS/TANF/FDPIR. For the sake of simplicity, we assume in the table that all those receiving 
FS/TANF/FDPIR have incomes of no more than 130 percent of poverty, and so are subgroups of 
the first column in the table (and denoted A, B’, F’, J’, and N’ ). 
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TABLE II. 1 



STUDENTS’ POSSIBEE COMBINATIONS OE INCOME AND CERTIEICATION 

STATUS 



Certification Status 


<= 130% EPE^ 


Income-Eligibility Status 

131 to 185 >185 Percent 

Percent EPE EPE 


Total 


Directly Certified'’ 


A 


— 


— 


A 


Certified Eree by 
Application 


B 


C 


D 


E 


Certified Reduced-Price 


E 


G 


H 


I 


Not Certified (Paid) 


J 


K 


E 


M 


Total 


N 


O 


P 


Q 



‘‘This column potentially includes both students receiving food stamp or Temporary Assistance 
for Needy Eamilies (ES/TANE) benefits and students not receiving these benefits. Eor 
simplicity, we assume that no ES/TANE recipients have incomes above 130 percent of the 
federal poverty level (EPE). We denote the groups of students on ES/TANE in the various 
certification categories as: A (directly certified, where all directly certified students are 

assumed to be on ES/TANE), B’ (certified free), E’ (certified RP), J’ (paid), N’ (total). 

'’We assume that all directly certified students have incomes below 130 percent of the EPE. 
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procedures tested in the NSLP Application/Verification Pilot Projects affect this percentage? 
Referring to measures of accuracy in Table II. 1, the percentage of students certified for free 
meals who are accurately certified is (A+B)/(A+B+C+D). Accuracy depends on (1) the number 
of eligible students who are certified for free meals (A + B) and (2) the number of ineligible 
students who are not certified for free meals (C + D). The accuracy rate will increase as the 
number of eligible students who are certified increases, and it will increase as the number of 
ineligible students certified decreases. Basically, the goal is that fewer ineligible students and 
more eligible ones are approved. 

Our approach to the analysis recognizes that improving certification accuracy requires 
different responses from two key groups of students: to maximize the improvement in accuracy 
income-ineligible students must reduce their likelihood of becoming certified, while income- 
eligible students must not reduce their likelihood of becoming certified. We recognize the 
possibility that the pilot demonstrations may induce different responses among the two groups by 
estimating certification rates separately for ineligible students and eligible students. If Up-Front 
Documentation and Graduated Verification were successful in reducing certification among 
ineligible students, we refer to this effect as evidence of deterrence. However, to the extent that 
these interventions led to a reduction in certification among eligible students, we would refer to 
this effect as evidence of barriers. Recent policy discussions have focused on the ratio of the 
number of students approved for free meals nationally according to FNS administrative data 
(A-i-E in Table II. 1) to the number of students nationally in households with income less than 
130 percent of the poverty level (N in Table II. 1). A weakness in the available national data is 
that they do not allow us to examine separately the status of eligible and ineligible students. 
Without information on the percentage of eligible students certified (or the percentage of 
ineligible students certified), the ratio of the number of students approved nationally to the 
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number eligible nationally provides no insight into the percentage of students nationally who are 
certified inaccurately. If, for example, the number approved and the number eligible were equal 
to one another, this could either result from perfect accuracy (with all eligible students and no 
ineligible students certified) or from a highly inaccurate system (if a large number of ineligible 
students were certified but an equally large number of eligible students were not certified). The 
study survey provided the critical information on whether eligible/ineligible students are certified 
for the pilot and comparison districts, and our outcome measures use this information. 

In this report, we present data on two measures of certification deterrence (Table II.2): 
(1) the percentage certified for free meals among those with incomes above 130 percent FPL 
(CD_1), and (2) the percentage certified for free or reduced-price meals among those with 
incomes above 185 percent FPL (CD_2). 

Barriers are assessed through two measures for students with incomes below 130 percent 
FPL: the first measure (CB_la) counts as appropriately certified only those students correctly 
certified for free meals, and the second measure (CB_2a) counts as appropriately certified those 
with incomes below 130 percent FPL who are correctly certified for free meals and those who 
are incorrectly certified for reduced-price meals. The two measures treat differently those 
students who have incomes below 130 percent FPL and were approved for reduced-price meals. 
The first measure considers this group as not appropriately certified, and thus potentially subject 
to a barrier. The second measure considers this group as appropriately certified, and thus not 
subject to a barrier. The third barrier measure (CB_3a) assesses barriers for students with 
incomes below 185 percent FPL. The fourth measure (CB_4a) assesses barriers for a specific 
subgroup of interest: students who are recipients of TANF or food stamps but were not directly 
certified for free meals. Each of these four measures is presented separately for students who 
were not directly certified for free meals (CB_la - CB_4a), and for all students (including those 
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OUTCOME MEASURES USED IN THE IMPACT ANALYSIS 
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Measures of Certification Targeting Efficiency: 

Estimates excluding directly certified 
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directly certified) (CB_lb - CB_4b). Together, these measures support a broad examination of 
whether and to what extent the pilot procedures may have created barriers to the certification of 
some eligible students. 

The certification accuracy measures combine information on deterrence and information on 
barriers to show the percentage of eligible students among those who are certified. We include 
one measure that examines accuracy among students approved for free meals (CA_la) and three 
measures that examine accuracy among students approved for free or reduced-price meals 
(CA_2a - CA_4a). The three measures that examine accuracy for students approved for free or 
reduced-price meals adopt different perspectives on whether approval of students for a benefit 
other than the one they were eligible for is appropriate. The first measure of free and reduced- 
price meal accuracy (CA_2a) counts as accurate only those receiving the correct benefit. The 
second measure (CA_3a) counts as accurate those receiving the correct benefit or who are 
certified for a lower level of benefits than they are entitled to. It is the percentage of 
free/reduced-price approved students who are not overcertified. Finally, the third measure of 
free and reduced-price meal accuracy (CA_4a) classifies as accurate any student who is certified 
for free or reduced-price meals even if they are overcertified but have income less than 185 
percent FPL. This measure counts as accurate students in Group C in Table II. 1, who are eligible 
for reduced-price meals but certified for free meals. It is the percentage of free and reduced- 
price approved students whose income is not over 185 percent FPL. While this set of measures 
does not cover all of the possible definitions of accuracy, it aims to cover a broad range of the 
possible definitions in order to identify whether the pilot projects may have affected accuracy for 
some students. The same set of measures is calculated including directly certified students 
(CA_lb - CA_4b). 
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Finally, for targeting efficiency, we present three measures using the three variants of the 
definition of accuracy for students approved for free and reduced-price meals (CTE_la, CTE_2a, 
CTE_3a). Each measure of targeting efficiency differs from the corresponding measure of 
accuracy by including students paying full price in the numerator and students above 185 percent 
EPE in the denominator. 

As is clear from Tables II. 1 and II.2, measuring the outcomes of interest requires 
information on students’ certification status and household income. We obtained information on 
certification status from the administrative data provided by districts — the lists of students in 
different certification categories. 

Household income information was obtained from the telephone and in-home surveys, as 
described in Section A. 3. The data collection process, however, resulted in some cases in which 
households failed to provide complete income information. If the household provided no 
information on their income, they were dropped from the analysis file and the sample weights 
were constructed to account for the fact that these households were nonrespondents with respect 
to household income. However, there were two situations in which respondents provided partial 
income. Eirst, some respondents completed the telephone (part 1) interview, and provided 
information that their income was below 400 percent of poverty — thus making them a target for 
the in-home (part 2) interview — but failed to respond to the in-home interview. We refer to 
these as totally missing part 2 income. Second, other respondents completed both interviews, but 
failed to provide data on one or more items during the in-home interview necessary to calculate 
total household income. We refer to these cases as partially missing part 2 income. 

Of our total sample of 3,685 students, part 1 interviews were completed for 3,020, of which 
401 households reported income above 400 percent of poverty so that a part 2 interview was not 
attempted. Among the 2,619 students for whom part 2 interviews were attempted, interviews 
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were completed in 2,125 cases. Thus, 494 cases were totally missing part 2 income. Among 
part 2 respondents, 1,880 provided complete income information and 245 were partially missing 
part 2 income. We estimate that 5 percent of the cases released for interview were ineligible, 
usually because the child was no longer enrolled in the district. Therefore, the overall 
unweighted percentage of cases with a completed part 2 interview or a part 1 interview 
indicating income above 400 percent of poverty was 72 percent. The percentage with a part 1 

o 

interview was 86 percent. 

In cases with totally missing part 2 income (but with valid part 1 income) and in cases with 
partially missing part 2 income, we imputed total household income. The decision to impute 
income in these cases as opposed to dropping them from the analysis file was based on two 
factors. First, we had substantial — albeit not complete — information on household income for 
these cases, so that the imputed value of income was based to a large degree on actual income 
information reported by the households. Second, we felt that dropping these cases would 
potentially result in our estimates being biased if the households dropped from the analysis were 
different in key ways from households included in the analysis. 

To impute household income when part 2 income was totally missing, we used information 
on the reported part 1 income of these households along with information on the relationship 
between part 1 and part 2 income provided by sample households with complete income 
information from both sources. In particular, we used the complete income information from 
this group of households to estimate a set of logistic regression models predicting households’ 
part 2 income-eligibility category (free/RP/paid) based on reported part 1 income and other 
household characteristics. We used the parameters of this model along with the reported part 1 

^See Chapter IV of Volume II. 
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income information and other household characteristics reported on the part 1 survey to impute 
the part 2 income-eligibility status of households that did not complete the part 2 interview. 

For cases with partially missing part 2 income, we first identified what components of 
income were missing. Households with partially missing income information may have failed to 
report income from any of a number of different income sources brought into the household by 
any of a number of different household members. Once the missing source/household member 
was identified, we imputed its value based on typical values for that source and type of 
household member among respondents for whom we had valid data. Specifically, four 
imputation methods were used: (1) data edits, (2) median replacement, (3) Bernoulli random 
imputations, and (4) sequential hot-deck imputations. Once these missing sources were imputed 
for a given household, we combined their values with the valid part 2 income information they 
provided to generate an overall estimate on income (and income eligibility category) for that 
household. 

For more details on the process by which we imputed household income for cases with 
partially or totally missing part 2 income, see Volume II, Chapter VIII. That chapter also 
contains the results of sensitivity analyses we conducted to assess the degree to which the 
specific imputations procedures we used influenced the estimated impacts on the key outcomes 
of interest in the analysis. 

b. Methods for Obtaining Impact Estimates 

We used regression methods to estimate how districts’ mean levels of outcomes related to 
deterrence and barriers were influenced by the pilot demonstration.^ These regression methods 



^ For additional details on the methods described in this section, see Volume II, Chapter IX. 
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allowed us to control for exogenous household- and student-level characteristics in estimating 



pilot impacts. 

The general model used to estimate impacts on deterrence/barriers was: 



K K 



(1) 


yi = 


c + Xfi + '£dfiP,+'£a,[DP,*P,]+e, 

j=2 k=l 


where: 


ji = 


outcome of interest for student i 






vector of characteristics of student i hypothesized to affect outcome y. 




DPij 


= binary indicator of whether student i was in the jth pilot-comparison 
district pair 




Pi = 


binary indicator of whether student i was in a pilot district 




ei = 


random error term 



The outcomes of interest for deterrence/barriers are shown in Table II.2. The model, which 
was estimated separately for Up-Front Documentation and Graduated Verification districts, was 
estimated using ordinary least squares (OLS). Essentially, the outcome was regressed on student 
characteristics, a set of binary variables (called district pair variables) that each represented a 
pilot district along with its matched comparison district, and a set of variables formed by 
interacting a district pair variable with a binary variable indicating whether the student’s district 
was a pilot district. The coefficients on these interaction terms {ai through uk , where K 
represents the number of pilot districts that implemented Up-Front Documentation or Graduated 
Verification) represent the impacts of the demonstration in each of the pilot districts. The mean 
value of these coefficients is our estimate of the overall impact of Up-Front Documentation or 
Graduated Verification on the outcome of interest. 

To estimate impacts on outcomes related to deterrence, we based the estimation on samples 
of students ineligible for free meals or for free/reduced-price meals. To estimate impacts on 
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outcomes related to barriers, we used samples of students eligible for benefits, excluding directly 
certified students. We could not include directly certified students in the regression models 
because we did not collect survey data from these students and thus had no information on their 
characteristics. 

To generate estimated impacts on measures of barriers among all eligible students 
(including those directly certified), we relied on the assumption that all directly certified students 
were eligible for free meals at the time the survey data were collected. By using this assumption, 
along with information on the proportion of students in each district who were directly certified, 
we translated our estimate of the impact of Up-Front Documentation or Graduated Verification 
on barriers among non-directly certified students into an estimate of this impact among all 
eligible students. 

Estimating impacts on accuracy and targeting efficiency proceeded differently from 
estimating impacts on deterrence/barriers. Measures of deterrence and barriers are defined 
according to the certification status of a sample of either eligible or ineligible students. Thus, 
impacts on these outcomes could be estimated using a regression model in which the dependent 
variable was the student’s certification status and the explanatory variables included the pilot 
status of the student’s district (along with various other variables). By contrast, measures of 
accuracy are defined according to the eligibility status — that is, the household income level — of 
a sample of certified students. We did not estimate a regression model in which income 
eligibility status (free eligible/RP eligible/not eligible) was a dependent variable and pilot status 
and other student characteristics were explanatory variables because this would have implied that 

“’We used a similar set of procedures to translate our estimates of impacts on accuracy and 
targeting efficiency among non-directly certified students to estimates on these outcomes among 
all groups of students that included those directly certified. 
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we believed that pilot status could potentially influence students’ income level, and we did not 
believe this. Furthermore, the sample of such a regression would have been endogenous, since 
the pilot demonstration is hypothesized to influence students’ certification status, and the 
regression model would not have produced unbiased estimates of the impact of Up-Front 
Documentation or Graduated Verification on accuracy.'' 

To estimate impacts of Up-Front Documentation and Graduated Verification on accuracy 
and targeting efficiency, we used the results of the estimated impacts on deterrence and barriers, 
along with basic probability theory. In particular, the estimated impacts on deterrence and 
barriers reveal how the pilot interventions affected certification rates among ineligible students 
(deterrence) and among eligible students (barriers). To examine accuracy and targeting 
efficiency, we need to know how the pilots affected the proportion of certified students eligible 
for benefits (to address accuracy) and the proportion of all students with consistent certification 
and eligibility statuses (to address targeting efficiency). So long as we know the proportion of 
students eligible versus ineligible, we can derive the impact on eligibility among certified 
students based on the estimated impacts on certification among eligible and ineligible students. 
Volume II, Chapter IX shows precisely how these estimated impacts on accuracy and targeting 
efficiency have been derived. 

5. Limitations of the Study 

Like all studies, this one is subject to several limitations that should be clearly understood so 
that the findings are used appropriately. We note these here. 



"a similar set of problems with the targeting efficiency outcome measures prevented us 
from estimating targeting efficiency regression models. 
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a. Comparison Group Design 

First, the impacts of the pilot projects on deterrence, barriers, and accuracy are measured by 
comparing the outcomes of families in the pilot districts with the outcomes of families in 
comparison districts. The identification of suitably matched comparison districts was done with 
great care, and we controlled statistically for a range of personal and family characteristics that 
could influence the outcomes. As shown in the next chapter, the comparison district matching 
process produced a set of comparison districts and a sample of families with characteristics 
similar to those of the pilot districts. However, while, in our judgment, these comparison 
districts provide a reasonable basis for measuring net impacts of the pilot interventions, there 
remains an element of uncertainty about the quality of the benchmark that we cannot quantify. 

b. Sample Size Limitation 

Second, as in most studies, resource constraints limited the size of the samples that it was 
possible to interview. For most outcomes, the study samples are of sufficient size to give us 
confidence that if the pilot projects caused an impact of a policy relevant magnitude — such as an 
impact of 20 to 30 percent of the mean outcome in comparison districts, our sample would 
provide a high likelihood of detecting the impact. However, for some variables, limits on sample 
sizes place an important constraint on our ability to detect significant demonstration effects. For 
example, in examining deterrence effects, while the sample can detect impacts of a small 
absolute magnitude, these impacts are fairly large in relative terms. In Up-Front Documentation 
pilots, our sample has an 80 percent chance of detecting an impact of two percentage points 
using a two-tailed test with a 95 percent confidence level (such as a reduction in the percentage 
of students certified among students with incomes exceeding 185 percent of the federal poverty 
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level from 4 percent to 2 percent). However, an impact of two percentage points would represent 
a reduction of 50 percent relative to the mean of the outcome in the comparison districts. 

c. Issues of Generalizability 

Third, the small number of demonstration sites and the voluntary nature of the decision to 
participate in the pilot projects necessarily limit our ability to draw conclusions about what 
would happen if the policies tested were to be implemented nationwide. Just nine districts 
included in the study implemented Up-Front Documentation and just three implemented 
Graduated Verification. Furthermore, these districts were part of a very small group nationwide 
that volunteered to test new procedures designed to improve the accuracy of the process for 
administering NSLP certification. We document in Chapter III how these districts as a group 
compare with the nation as a whole in terms of some readily observable characteristics of the 
districts. However, one can only speculate on how these districts differed from others 
nationwide in terms of unobservable characteristics that are likely to affect the outcomes of 
interest in the evaluation. 

B. IMPLEMENTATION OF THE PILOT PROJECTS AND THE EVALUATION 

In January 2000, FNS published a Federal Register notice inviting SFAs to participate in 
one of four pilot projects that could operate for the three-school-year period from fall 2000 to 
spring 2003 {Federal Register, January 21, 2000). In response to the request for applications, at 
the start of school year 2000-2001, 10 SFAs began operating the Up-Front Documentation 
Model, 4 began operating the Graduated Verification Model, and 7 began operating the Verify 
Direct Certification Model (Table II.3). Two additional SFAs responded later and began 
operating the Up-Front Documentation Model at the start of school year 2001-2002. Thus, a 
total of 12 SFAs operated the Up-Front Documentation Model. 
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TABLE II.3 



DISTRICTS PARTICIPATING IN THE NSLP APPLICATION/VERIEICATION PILOT PROJECTS 



Project District 


Still Operating 
Pilot as of Pall 
2002 


Included in 
Study 


Reason for Exclusion 


Up-Front Documentation 
Districts 

Glenview, IL 


Yes 


No 


District had been doing 100 percent 


Kismet, KS 


No 


No 


verification for several years 
Dropped out after year 1 


Morenci, AZ 


Yes 


No 


No match possible 


Blue Ridge, PA 


Yes 


Yes 




East Stroudsburg, PA 


Yes 


Yes 




Pleasant Valley, PA 


Yes 


Yes 




Stroudsburg, PA 


Yes 


Yes 




Maplewood, OH 


Yes 


Yes 




Salem, OH 


Yes 


Yes 




Creve Coeur, IL 


Yes 


Yes 




Oak Park, IL 


Yes 


Yes 




Williamson, TN 


Yes 


Yes 




Graduated Verification 
Districts 

Dilworth-Glyndon-Eelton, MN 


Yes 


Yes 




Dunkirk City, NY 


Yes 


Yes 




Grandview, MO 


No 


Yes 


District withdrew from pilot after year 2 


St. Mary’s, Paterson, NJ 


Yes 


No 


but remained in evaluation 
District not included in evaluation due to 


Verify Direct Certification 
Districts 

Alma, MI 


Yes 


No 


implementation difficulties 
Evidence in “National School Lunch 


Bessemer, AL 


Yes 


No 


Program Application/ Verification 


Blount County, TN 


Yes 


No 


Pilot Project; Report on Pirst Year 


East Baton Rouge, LA 


Yes 


No 


Experience” indicated high rate of 


Kenai Peninsula, AK 


Yes 


No 


certification accuracy among 
directly certified students 


Middlebury, IN 


Yes 


No 


Perry County, AL 


Yes 


No 
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Of these three models, the Verify Direct Certification Model was excluded from the 
evaluation. Under this model, students who were approved through direct certification were 
subject to the standard verification procedures similar to those used with households approved on 
the basis of an application. Analysis of the first-year experience of these seven SFAs showed 
that very few directly certified students become ineligible later in the same school year. On the 
basis of this finding, FNS decided not to include the Verify Direct Certification pilots in the 

1 9 

evaluation reported herein. 

Nine of the 12 SFAs that tested Up-Front Documentation and 3 of the 4 that tested 
Graduated Verification are included in the evaluation described in this report. Reasons for 
excluding districts that operated the demonstration were diverse. Morenci, Arizona, was 
excluded because we could not identify a credible comparison district. St. Mary’s, Paterson, 
New Jersey, was excluded because it had experienced difficulty implementing the pilot 
procedures, and therefore would not have provided a valid test of pilot procedure impacts. 
Glenview, Illinois, was excluded because it had been verifying all cases soon after application 
for approximately 20 years. Kismet, Kansas, was excluded because it stopped using 
demonstration procedures after one year of operation. 

This evaluation analyzes program operations and impacts in 12 of the pilot SFAs during a 
specific period — the third and final year of the planned implementation period, school year 
2002-2003. Table II.4 shows the history of pilot project implementation in these 12 SFAs, as 
well as which districts used direct certification during which years before and during the pilot 
implementation period. Students certified for free meals by direct certification are specifically 
not subject to the demonstration procedures. Furthermore, as noted above, these directly 



*^See U.S. Department of Agriculture 2002. 
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PERIOD OF OPERATION OF THE NSLP APPLICATION VERIFICATION PILOT PROJECTS AND 
USE OF DIRECT CERTIFICATION BY PILOT SFAs INCLUDED IN THE EVALUATION 
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Dunkirk City, NY No No Yes Yes Yes Yes Yes Yes Yes Yes 

Rounds of Verification 13 2 

Grandview, MO No No Yes Yes No Yes Yes Yes Yes Yes 

Rounds of Verification 3 3 1 




certified students have been excluded from the study’s student- and family-level data collection 
and much of the analysis. However, a district’s decision whether or not to use direct certification 
could have affected the average characteristics of students who are included in the study’s 
population of interest. Three SFAs in the evaluation stopped using direct certification and two 
started using it during the course of the pilot project (Table II.4). 

Several special situations are relevant to understanding and assessing the evaluation 
findings. First, as shown in Table II.4, two Up-Front Documentation SFAs included in the study 
(Pleasant Valley and Stroudsburg, Pennsylvania) began pilot operations in 2001-2002 and were 
operating the demonstration for their second year, rather than their third year, when the 
evaluation was conducted. Second, one Graduated Verification SFA included in the evaluation 
(Grandview, Missouri) did not operate under the pilot rules during the third pilot year. 
Grandview withdrew from the pilot project as of June 30, 2002, because, according to the 
superintendent, pilot operation had reduced the number of Grandview students approved for free 
and reduced-price meals in the first two pilot years, which had adversely affected the district’s 
receipt of state funding to an unacceptable degree.'^ 

With support of Grandview district officials, we decided to keep Grandview in the study. 
Because the extensive verifications done in the two prior years of full pilot operations and 
implementation of the requirement that households terminated provide documentation when 
applying in the next school year, we judged that whatever deterrent and barrier effects the pilot 
had would have been observed in school year 2002-2003. However, some families could have 
been affected by the district’s decision to withdraw from the pilot, because Grandview did not 

I T 

Personal communication from Paul Strasberg regarding letter from Grandview 
Superintendent John Martin to Paul Strasberg, April 2002. 
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require fa mi lies whose students’ benefits were reduced or terminated in school year 2001-2002 
to provide documentation of their income when applying in school year 2002-2003, as the pilot 
protocol intended. 

A third group of special situations that affects interpretation of the findings is coverage of 
the district. In most pilot and comparison sites, the entire district is part of the study, but there 
are two exceptions. In one pilot site — Williamson County, Tennessee — the pilot project 
operated in 9 of the 29 schools in the district; it did not operate in the other 20 schools. 
However, the corresponding comparison site — Wilson County, Tennessee — includes the entire 
school district.'"^ In one pilot comparison pair — Oak Park pilot site and its comparison. Valley 
View — the pilot site is a single-school high school district, while the comparison site is one high 
school within a much larger district. Accordingly, in this site pair we were unable to achieve full 
comparability in this key feature of the administrative setting. 



'’^We decided to include the entire district because we lacked school-level data with which to 
match a subset of the Wilson County schools to the Williamson pilot schools. 
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III. CHARACTERISTICS OF THE PILOT AND COMPARISON DISTRICTS 



One objective of this study is to describe the characteristics of the students, families, and 
districts included in the evaluation. This descriptive analysis addressed two groups of questions: 
(1) What do the demonstration districts and their populations look like and how do they compare 
with the nation as a whole? and (2) Do the comparison districts provide a credible benchmark for 
the pilots?^ To address these questions, we used individual-level interview data on the families 
in our sample and district-level data available from the National Center for Education Statistics 
(NCES), the U.S. Census Current Population Survey (CPS), and ENS. 

We first describe the characteristics of the Up-Eront Documentation and Graduated 
Verification pilot districts and compare them to the nation as a whole. We then present the pre- 
pilot differences between the pilot and comparison districts. Einally, we discuss differences 
between the pilot and comparison district students and families in our sample. 

A. CHARACTERISTICS OF THE PILOT DISTRICTS 

SEAs in the pilot districts volunteered to participate in the NSEP Application/Verification 
Pilot Projects. Since pilots were not specifically selected to be representative of the nation as a 
whole, the pilot sites may differ in some ways from the average public school district nationally. 
Table III.l shows selected characteristics of the Up-Eront Documentation and Graduated 
Verification pilot districts and all public school districts nationwide. This table and the 
discussion in this section are based on pre-pilot data, specifically the NCES Common Core of 
Data (CCD) from the 1999-2000 school year and 1999 poverty data from the CPS. 



Volume II of this report addresses a third related issue, the comparison of survey 
respondents to nonrespondents to assess whether the sample successfully interviewed is similar 
to the sample selected. 
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TABLE III. 1 



PRE-PILOT CHARACTERISTICS OF UP-FRONT DOCUMENTATION AND GRADUATED 

VERIFICATION PILOT DISTRICTS 



Characteristics 


Pilot Districts” 

Up-Front Graduated 

Documentation Verification 


Nationwide 

Districts 


Region 


Northeast 


0.0 


33.3 


14.2 


Mid- Atlantic 


44.4 


0.0 


9.7 


Midwest 


44.4 


33.3 


23.2 


Mountain Plains 


0.0 


33.3 


19.5 


Southeast 


11.1 


0.0 


7.0 


Southwest 


0.0 


0.0 


13.4 


West 


0.0 


0.0 


13.1 


Urbanicity 


Central Cities 


0.0 


0.0 


8.2 


Non-Central City Areas within MSAs 


66.7 


100.0 


40.7 


Non-MSAs 


33.3 


0.0 


51.1 


District Size’’ 


Number of schools (mean) 


7.6 


6.7 


5.6 


Number of students (mean) 


4,874.7 


2,612.3 


2,925.1 


Student Enrollments (Percentages) 


0 to 500 


0.0 


0.0 


38.3 


500 to 1,000 


11.1 


0.0 


15.5 


1,000 to 2,500 


22.2 


66.7 


21.8 


2,500 to 5,000 


33.3 


33.3 


12.9 


5,000 to 10,000 


22.2 


0.0 


6.4 


10,000 to 25,000 


11.1 


0.0 


3.6 


25,000-t 


0.0 


0.0 


1.5 


Title I Eligible Schools (Percentage) ’’ 


40.6 


57.8 


54.8 


Schoolwide Title I Schools (Percentage) ’’ 


0.0 


6.7 


18.7 


NSLP Certification Status (Percentages)’’ 


Free 


16.6 


33.3 


25.7 


Reduced-price 


6.2 


9.0 


7.7 


Percentage of School-Age Children Below 100 


Percent of FPL in 1997 


10.4 


19.4 


17.0 


Percentage of School-Age Children Below 100 


Percent of FPL in 1999 


9.3 


22.4 


14.6 


Race/Ethnicity (Percentages)” 


White, non-Hispanic 


88.4 


65.8 


79.4 


Black, non-Hispanic 


6.9 


18.3 


8.8 


Hispanic 


3.4 


14.1 


8.3 


Native American 


0.2 


1.3 


2.8 


Asian, Pacific Islander 


1.1 


0.5 


1.6 


Eimited English Proficiency (Percentage) ‘ 


0.2 


5.5 


4.0 


Sample Size 


9 


3 


16,887 
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TABLE III. 1 (continued) 



Source: 1999 Poverty data from the 2000 Census; all other data from the NCES Common Core of Data (1999-2000 
school year). 

Note: Each district is weighted equally in each column of this table, thus the characteristics presented are 

representative of the average district, not the average student. 

‘‘Characteristics are presented for the subset of pilot districts included in this evaluation. Morenci, Glenview, and St. 
Mary’ s are excluded. 

'’Includes all schools and students in the district, regardless of pilot participation. Only 9 of the 29 schools in Williamson 
participated in the pilot. 

“Data not available for all sites: race and Title 1 information was missing for one Up-Front Documentation district, 
certification data were missing for two districts, and limited English proficiency data were missing for five districts. 
Complete data were available for all Graduated Verification sites. 

‘'Percentage of children ages 5 to 17 with household income below 100 percent of the federal poverty level. This is a 
lower threshold than is used to determine eligibility for free NSLP meals. 
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The pilot sites in our study were eoneentrated in the Midwest and Mid-Atlantie states. Four 
were in Pennsylvania, two in Illinois, and two in Ohio; Minnesota, Missouri, New York, and 
Tennessee eaeh eontained one pilot study site. Both Up-Front Doeumentation and Graduated 
Verification pilot sites were disproportionately located in non-central city locations within 
metropolitan areas, compared with public school districts nationwide. 

Up-Front Documentation pilots were also somewhat larger than districts nationally, but no 
very large districts were represented in the demonstration. Pilot districts included more schools 
(7.6, on average, compared to 5.6 nationwide) and more students (almost 4,900, compared to a 
national average of about 2,900). Most pilot districts served between 1,000 and 10,000 students, 
while more than half of public school districts across the country included fewer than 1,000 
students. Only one of the pilot districts contained more than 10,000 students, and none contained 
more than 25,000. While fewer than 2 percent of public school districts nationally included 
more than 25,000 students, these very large districts enrolled about one-third of all public school 
students nationally. 

Up-Front Documentation districts had lower poverty rates among school-age children than 
all districts nationwide (9 percent, on average, compared to 15 percent), and the pilots included 
fewer Title I schools. Correspondingly, a smaller percentage of students in Up-Front 
Documentation pilot districts were certified to receive free or reduced-price meals in the pre- 
pilot period than in the country as a whole (23 percent, compared to 33 percent). 

2 

The largest Up-Front Documentation pilot district, Williamson, contained just over 20,000 
students, but only 9 of the 29 schools in that district participated in the demonstration. 

^Title I is a federal program to assist economically and educationally disadvantaged students 
to achieve academically at the same level as their peers. Schools are eligible to receive Title I 
funding based on the level of poverty among the students served. In schoolwide Title I schools. 
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Finally, Up-Front Documentation pilot districts served lower proportions of minority 
students and students with limited English proficiency (LEP) than the national average. Eor 
example, only 3 percent of students in pilot districts were Hispanic, compared to 8 percent in 
other districts. 

Graduated Verification pilot districts were more similar to the nation as a whole in several 
respects. Eor example, with 6.7 schools and about 2,600 students in the average Graduated 
Verification pilot, they were closer than Up-Eront Documentation districts to the national 
average in size. Again, however, the very largest public school districts were not represented in 
the Graduated Verification pilot districts, the largest of which enrolled only about 4,200 students. 

Several indicators suggest that Graduated Verification districts were somewhat less affluent 
than the typical district nationally. Their percentage of school-age children in families with 
income below 100 percent of the federal poverty level (22 percent) was higher than the national 
average. Graduated Verification districts also had higher-than-average rates of NSEP 
certification (33 percent certified free and 9 percent reduced-price), and somewhat higher 
percentages of Title I eligible schools (58 percent). In addition, they included higher-than- 
average proportions of African American and Hispanic students and EEP students. 

The voluntary nature of participation in the demonstration and the small number of pilot 
sites, by themselves, limit our ability to draw conclusions from the demonstrations about the 
likely effects of wider implementation of the pilot procedures. The differences noted here 
underscore the need for caution in applying the findings from the pilots to the national program. 



(continued) 

Title I funds can be used for all students. In other schools. Title I funds can be used only for 
students who meet certain eligibility criteria. 
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B. COMPARING PILOT AND COMPARISON DISTRICTS DURING THE PRE- 
PILOT PERIOD 

The characteristics of Up-Front Documentation comparison districts matched those of the 
corresponding pilots closely."^ The Graduated Verification comparison districts differed a bit 
more from their pilots, but were nevertheless generally similar. Table III.2 shows selected 
characteristics of the Up-Front Documentation and Graduated Verification pilot and comparison 
districts during the pre-pilot period. For most of the characteristics included in this table and 
discussed in this section, data on the year just before the pilot began were available in summer 
2002 and used in selecting comparison sites. ^ However, the most recent CPS poverty data 
available at the time of comparison site selection covered 1997, so for that characteristic we 
present both 1997 data (to assess the quality of the matching process) and 1999 data (to compare 
baseline characteristics). 

In Up-Front Documentation comparison districts, the average number of schools and 
students, the poverty rates among school-age children, and the race/ethnicity of students were all 
quite similar to those characteristics in the pilot districts.^ The percentages of students certified 



"^Our comparison site selection process is summarized in Chapter II and described in detail 
in Volume II, Chapter II. 

^Specifically, the data sources used in selecting comparison sites were 1999 enrollment and 
NSLP certification and participation data from the FNS Minimum Data Set, 1997 poverty data 
from the CPS, and remaining data from the NCES CCD on the 1999-2000 school year. Chapter 
X of Volume II presents more recent data and discusses how pilot and comparison sites changed 
over time. 

^Although the poverty rates among school-age children in pilot and comparison districts 
were similar in both 1997 and 1999, they were closer in 1997. The poverty rate declined 
between 1997 and 1999 in both pilot and comparison districts, but the reduction was somewhat 
greater in pilots, resulting in a slightly larger gap between pilots and comparisons than was 
observed at the time of comparison site selection. 
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TABLE III.2 



PRE-PILOT CHARACTERISTICS OF UP-FRONT DOCUMENTATION AND 
GRADUATED VERIFICATION PILOT AND COMPARISON DISTRICTS 





Up-Front Documentation 




Graduated Verification 


Characteristics 


Pilot 

Districts 


Comparison 

Districts 


Difference 


Pilot 

Districts 


Comparison 

Districts 


Difference 


District Size 


Number of schools (mean)^ 


7.6 


7.0 


0.6 


6.7 


9.5 


-2.8 


Number of students (mean)^ 


4,874.7 


5,012.4 


-137.7 


2,612.3 


4,575.2 


-1,962.9 


Student Enrollments (Percentage)^ 


0 to 500 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


500 to 1,000 


11.1 


11.1 


0.0 


0.0 


33.3 


-33.3 


1,000 to 2,500 


22.2 


33.3 


-11.1 


66.7 


0.0 


66.7 


2,500 to 5,000 


33.3 


22.2 


11.1 


33.3 


0.0 


-33.3 


5,000 to 10,000 


22.2 


11.1 


11.1 


0.0 


66.7 


66.7 


10,000 to 25,000 


11.1 


22.2 


-11.1 


0.0 


0.0 


0.0 


25,000+ 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


Title I Eligible Schools (Percentage)'’ 


40.6 


45.1 


-4.5 


57.8 


73.2 


-15.4 


Schoolwide Title 1 Schools (Percentage)'’ 


0.0 


6.3 


-6.3 


6.7 


22.5 


-15.8 


NSLP Certification Status (Percentage) 


Free 


16.5 


16.5 


0.0 


34.9 


32.2 


2.7 


Reduced-price 


5.9 


5.9 


0.0 


9.6 


9.6 


0.0 


NSLP Participation (Percentages)^ 
Average Daily Participation Rate 


Among all students 


42.9 


45.8 


-2.9 


66.2 


60.8 


5.4 


Among students certified free 


74.5 


76.1 


-1.6 


80.8 


78.2 


2.6 


Among students certified reduced-price 


70.8 


72.0 


-1.2 


69.5 


73.3 


-3.8 


Among students not certified 


35.8 


36.6 


-0.8 


56.1 


47.6 


8.5 


Percentage of School-Age Children Below 


100 Percent of FPL in 1997' 


10.4 


10.3 


0.1 


19.4 


18.8 


0.6 


Percentage of School-Age Children Below 


100 Percent of FPL in 1999' 


9.3 


9.9 


-0.6 


22.4 


16.4 


6.0 


Race/Ethnicity (Percentages)*^ 


White, non-Hispanic 


88.4 


88.7 


-0.3 


65.8 


70.6 


^.8 


Black, non-Hispanic 


6.9 


6.9 


0.0 


18.3 


24.0 


-5.7 


Hispanic 


3.4 


3.0 


0.4 


14.1 


4.0 


10.1 


Native American 


0.2 


0.0 


0.2 


1.3 


0.9 


0.4 


Asian, Pacific Islander 


1.1 


1.3 


-0.2 


0.5 


0.6 


-0.1 


Limited English Proficiency (Percentage)*^ 


0.2 


0.9 


-0.7 


5.5 


1.4 


4.1 


Sample Size'' 


9 


9 




3 


3 





Source: NCES Common Core of Data (1999-2000 school year); 1999 FNS Minimum Data Set; 2000 Census. 

Note: Each district is weighted equally in each column of this table; thus, the characteristics presented are representative of the average 

district, not the average student. 

^Includes all students and schools in the district, regardless of pilot participation. 

^Data not available for all sites: race and Title 1 information was missing for one Up-Front Documentation pilot/comparison pair, limited 
English proficiency data were missing for five pilot/comparison pairs, and NSLP participation data were missing for six pairs. Complete 
data were available for all Graduated Verification sites. 

‘^Percentage of children ages 5 to 17 with household income below 100 percent of the federal poverty level. This is a lower threshold than is 
used to determine eligibility for free NSLP meals. 

‘^Although two comparison districts, Breckenridge and Lake Park Audubon, were selected for the pilot district Dilworth-Glyndon-Felton, 
Minnesota, the comparison districts were treated as a single district as the comparison observation in the analysis. The simple mean of the 
relevant variables is used in computing the comparison. 



47 




to receive free and reduced-price meals were almost identical. The NSLP participation rates 
were also similar, although slightly lower in pilot districts. One difference was that Up-Front 
Documentation pilots had lower proportions of Title I schools than comparisons (41 percent 
Title I eligible schools, compared to 45 percent, and 0 percent schoolwide Title I schools, 
compared to 6 percent). 

Because the evaluation included only three Graduated Verification districts (compared with 
nine Up-Front Documentation districts), there was less chance for differences between each 
Graduated Verification pilot district and its comparison to be offset by a difference in the 
opposite direction in another pilot-comparison district pair. Thus, somewhat larger differences 
existed in the baseline characteristics of Graduated Verification pilot and comparison districts, as 
compared with Up-Front Documentation pilot and comparison districts. However, the overall 
characteristics of Graduated Verification pilot districts — particularly the baseline rates of free 
and reduced-price meal certification — were still quite similar to those of comparison districts. 

Relative to their comparison districts. Graduated Verification pilot districts had fewer 
schools (6.7, compared to 9.5) and lower average enrollment (about 2,600 students, compared to 
4,600). The 1997 data available at the time of comparison site selection showed similar rates of 
poverty among school-age children in pilot and comparison districts (about 19 percent). 
However, the gap in child poverty rates widened from 1997 to 1999, as the percentage of 
children in poverty in pilot districts rose to 22 percent while the rate in comparison districts fell 
to 16 percent. Pilots included lower proportions of Title I eligible schools (58 percent, compared 
to 73 percent) and included higher proportions of Hispanic students (14 percent, compared to 
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4 percent)/ Graduated Verification pilots had higher rates of participation in the NSLP than 
their comparison sites, especially among students not certified (56 percent, compared to 48 
percent). 

C. DIFFERENCES IN THE CHARACTERISTICS OF STUDENT FAMILIES IN THE 
PILOT AND COMPARISON SAMPLES 

Once the comparison sites were chosen, we selected a stratified sample of students from 

o 

each pilot and comparison district. The sample included students approved for free meals by 
application, students approved for reduced-price meals by application, and students not certified 
for free or reduced-price meals. Students directly certified for free meals were specifically 
excluded from the sample. This group was excluded because the demonstration was expected to 
have no effect on them, since directly certified students were not subject to the NSLP application 
and verification procedures that were being modified in the pilots. However, their exclusion 
renders the sample not representative of the district population as a whole (the population 
described in Tables III.l and III.2), since directly certified students are predominantly on the 
lower tail of the income distribution and may comprise a sizable portion of the lower tail.^ The 



When we selected comparison districts for the Graduated Verification pilot sites, we were 
aware that we would be unable to select a comparison district for Dunkirk, New York, that 
matched well in terms of the percentage of Hispanic students, since the percentage of Hispanic 
students in Dunkirk is much higher than in neighboring districts. After selecting Jamestown, 
New York, as a comparison district however, we employed a sampling strategy in which 
Hispanic students were oversampled. As a result, the difference in the percentage of Hispanic 
students among sample members in pilot and comparison districts (11.5 percent in pilot districts 
versus 8.6 percent in comparison districts) was smaller and not statistically significant. 

o 

Chapter III of Volume II describes the sample selection process in detail. 

^Districts can directly certify students who are receiving cash public assistance or food 
stamps. Use of direct certification among all of districts in school year 2002-2003 is shown in 
Table II.4. Comparison districts matched their pilot district in using or not using direct 
certification, except in Blue Ridge, Pennsylvania (which did not use direct certification) and 
Montrose, Pennsylvania (which used direct certification). 
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TABLE III.3 



CHARACTERISTICS OF SAMPLE MEMBERS IN UP-FRONT DOCUMENTATION AND 
GRADUATED VERIFICATION PILOT AND COMPARISON DISTRICTS 



Up-Front Documentation Graduated Verification 



Characteristics 


Pilot 

Districts 


Comparison 

Districts 


Difference 

(Standard 

Error) 


Pilot 

Districts 


Comparison 

Districts 


Difference 

(Standard 

Error) 


Household Size 

Number of household members (mean) 


4.2 


4.3 


-0.12 

(0.06) 


4.3 


4.5 


-0.17 

(0.10) 


Number of children in household (mean) 


2.2 


2.3 


-0.08 

(0.05) 


2.4 


2.5 


-0.09 

(0.09) 


Household Structure (Percentages) 
Two-parent household 


75.3 


77.8 


-2.57 

(2.03) 


67.3 


71.2 


-3.87 

(3.30) 


Single-parent household 


23.5 


20.8 


2.71 

(1.99) 


31.8 


26.4 


5.36 

(3.27) 


Other household structure 


1.2 


1.4 


-0.14 

(0.57) 


0.9 


2.4 


-1.49* 

(0.73) 


Survey Respondent’s Educational Attainment 
(Percentages) 

Lacks a high school diploma 


7.0 


10.1 


-3.19* 

(1.33) 


13.0 


11.8 


1.17 

(2.36) 


High school diploma only 


41.1 


41.1 


0.09 

(2.45) 


29.8 


33.4 


-3.66 

(3.43) 


Some postsecondary education but lacks a 
college degree 


26.9 


30.3 


-3.36 

(2.32) 


39.4 


40.7 


-1.27 

(3.68) 


College degree or more 


25.0 


18.6 


6.47** 

(2.05) 


17.9 


14.1 


3.75 

(3.00) 


Employment Status of Household Members 
Survey respondent is employed (percentage) 


70.8 


74.2 


-3.37 

(2.24) 


77.3 


73.8 


3.52 

(3.04) 


Number of employed adults in household 
(mean) 


1.5 


1.6 


-0.13** 

(0.04) 


1.5 


1.5 


0.05 

(0.06) 


Household Income (Percentages) 
Less than 130 percent of FPL“ 


19.9 


21.0 


-1.04 

(1.70) 


29.5 


29.1 


0.41 

(2.88) 


131 to 185 percent of FPL 


12.4 


13.2 


-0.79 

(1.62) 


15.9 


14.3 


1.56 

(2.54) 


186 to 400 percent of FPL 


35.2 


41.4 


-6.26* 

(2.44) 


35.8 


38.9 


-3.15 

(3.62) 
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TABLE III. 3 {continued) 



Up-Front Documentation Graduated Verification 



Characteristics 


Pilot 

Districts 


Comparison 

Districts 


Difference 

(Standard 

Error) 


Pilot 

Districts 


Comparison 

Districts 


Difference 

(Standard 

Error) 


More than 400 percent of FPL 


32.5 


24.4 


8.09** 

(2.27) 


18.8 


17.6 


1.18 

(3.12) 


NSLP Eligibility 


Eligible for free meals 


19.9 


21.0 


1.04 

(1.70) 


29.5 


29.1 


0.41 

(2.88) 


Eligible for free meals or reduced-price 
meals 


32.3 


34.2 


-1.83 

(2.36) 


45.4 


43.4 


1.97 

(3.62) 


Not eligible for free meals 


80.1 


79.0 


1.04 

(1.70) 


70.5 


70.9 


-0.41 

(2.88) 


Not eligible for free meals or reduced- 
price meals 


67.7 


65.8 


1.83 

(2.36) 


54.6 


56.6 


-1.97 

(3.62) 


Percentage Receiving Public Assistance 


Percentage receiving food stamp benefits 


6.9 


6.9 


-0.03 

(1.06) 


13.8 


13.2 


0.67 

(2.27) 


Percentage receiving TANE 


1.3 


1.6 


-0.34 

(0.57) 


5.4 


5.9 


-0.54 

(1.68) 


Percentage receiving other benefits’’ 


13.1 


9.4 


3.72 

(1.98) 


12.0 


13.5 


-1.44 

(2.44) 


Percentage Residing in Public Housing or 
Receiving Housing Subsidy 


2.5 


2.2 


0.37 

(0.65) 


8.0 


6.9 


1.04 

(1.62) 


Percentage Who Own Their Home 


77.3 


76.1 


1.23 

(1.98) 


65.1 


67.2 


-2.08 

(3.24) 


Vehicle Ownership (Percentage) 


86.0 


88.3 


-2.28 

(1.90) 


82.6 


83.5 


-0.87 

(3.04) 


Number of Vehicles Owned by All Household 
Members (Mean) 


1.8 


1.8 


-0.07 

(0.06) 


1.6 


1.6 


-0.02 

(0.09) 


Household Mobility 


Number of Times Respondent Has Moved 
During Past Two Years (Means) 


0.3 


0.2 


0.03 

(0.03) 


0.3 


0.3 


0.01 

(0.05) 


Has Moved and Changed School Districts 
During Past Two Years (Percentage) 


10.9 


9.3 


1.57 

(1.51) 


10.5 


8.5 


1.96 

(1.97) 


Race/Ethnicity (Percentages) 


White, non-Hispanic 


83.4 


84.5 


-1.10 

(1.71) 


63.3 


58.5 


4.83 

(3.13) 


Black, non-Hispanic 


6.7 


6.1 


0.68 

(1.18) 


18.8 


24.0 


-5.22* 

(2.27) 


Hispanic 


2.0 


2.8 


-0.80 

(.069) 


11.5 


8.6 


2.91 

(2.33) 


Native American 


0.7 


0.2 


0.50 

(0.35) 


0.5 


0.2 


0.33 

(0.29) 
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TABLE III. 3 {continued) 



Up-Front Documentation Graduated Verification 



Characteristics 


Pilot 

Districts 


Comparison 

Districts 


Difference 

(Standard 

Error) 


Pilot 

Districts 


Comparison 

Districts 


Difference 

(Standard 

Error) 


Asian, Pacific Islander 


0.4 


1.2 


-0.85 

(0.44) 


0.3 


0.5 


-0.19 

(0.39) 


Other 


2.7 


2.2 


0.44 

(0.78) 


0.3 


2.0 


-1.67* 

(0.73) 


Mixed race 


4.2 


3.1 


1.13 

(0.96) 


5.4 


6.3 


-0.98 

(1.97) 


English Primary Language Spoken at Home 
(Percentage) 


97.3 


97.1 


0.18 

(0.80) 


92.9 


90.5 


2.43 

(2.21) 


Grade Level of Child (Percentages) 
Grade 9 to 12 


37.6 


34.4 


3.20 

(2.17) 


29.9 


28.4 


1.53 

(3.46) 


Grade 6 to 8 


20.7 


25.0 


-4.29* 

(2.06) 


28.0 


23.5 


4.49 

(3.42) 


Grade 3 to 5 


20.9 


22.6 


-1.61 

(2.06) 


22.2 


20.9 


1.27 

(2.86) 


Grade 1 to 2 


14.1 


12.2 


1.92 

(1.70) 


12.7 


16.7 


-4.01 

(2.72) 


Kindergarten or Pre-K 


6.1 


5.8 


0.35 

(1.20) 


7.0 


10.4 


-3.43 

(2.27) 


Pre-Pilot Free or Reduced-Price Certification 
Status (Percentage)'^ 


24.1 


21.6 


2.89 

(2.08) 


38.2 


39.3 


-1.14 

(3.94) 


Sample Size 


950 


988 




530 


535 





Note: Each district is weighted equally in this table. Within districts, households are weighted to adjust for nonresponse. 

'“The lowest income category (less than 130% of FPL) includes families categorically eligible for free meals (based on receipt of TANF or 
food stamps or foster child status), regardless of their actual income. 

’’ Other benefits include Social Security, Unemployment Insurance, general assistance, housing assistance, or other public assistance. 

‘ Students in grades K-2 were excluded from this calculation, since must would not have been enrolled in school, and thus not had access to 
the NSLP, at baseline. Sample sizes for this statistic are 674 Up-Front Documentation pilot households, 744 Up-Front Documentation 
comparison households, 387 Graduated Verification pilot households, and 357 Graduated Verification comparison households. 

FPL = federal poverty level. 

*Significantly different from zero at the .05 level. 

**Significantly different from zero at the .01 level. 
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rest of this chapter focuses on the students in this sample and their families, using data collected 
through interviews with households. 

The characteristics of sampled families in the pilot and comparison sites were generally 
similar. Table III.3 shows key characteristics based on survey data. Many characteristics, 
including pre-pilot NSLP certification rates, home and vehicle ownership, household mobility, 
and primary language spoken at home, did not differ significantly between families in pilot and 
comparison sites. However, there were some differences between pilot and comparison districts. 
These differences are described below, separately for Up-Front Documentation and Graduated 
Verification districts. 

Up-Front Documentation 

Sample members in Up-Front Documentation pilots tended to have higher education levels 
than those in comparison families. More pilot district parents had a college degree or more (25 
percent, compared to 19 percent of comparisons), and fewer lacked a high school diploma (7 
percent, compared to 10 percent of comparisons). In addition, households in Up-Front 
Documentation pilot districts tended to have higher incomes, despite including fewer working 
adults on average (1.5, compared to 1.6). Pilot district households were more likely to have 
incomes above 400 percent of poverty (33 percent, compared to 24 percent), although the 
proportion below 185 percent of poverty was quite similar (32 percent, compared to 34 percent). 
This result was driven primarily by two very-high-income pilot sites: Oak Park and Williamson. 

Differences in the pre-pilot certification status of sample members in Up-Front 
Documentation pilot and comparison districts were not statistically significant, although a 
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slightly higher percentage in the pilot districts were certified in the pre-pilot year (24 percent, 
compared to 22 percent) 

Graduated Verification 

The Graduated Verification pilot districts and their comparisons districts were similar in 
terms of household size, educational attainment of the survey respondent, employment status, 
income distribution, and most other characteristics considered. However, Graduated Verification 
pilot district households were less likely to have a household structure other than one -parent or 
two-parent household (0.9 percent, compared to 2.4 percent) and were more likely to be single- 
parent households (32 percent, compared to 26 percent) than comparisons (although the latter 
difference is only statistically significant at the 10 percent level). Fewer students in pilot districts 
were African American (19 percent, compared to 24 percent of comparisons), although these 
racial differences were due to differences in one pilot/comparison site pair. 

The pre-pilot free or reduced-price certification rates of students in Graduated Verification 
pilot and comparisons sites were nearly identical, at 38 percent in pilot districts and 37 percent in 
comparison districts.'' This difference was not statistically significant. 

D. SUMMARY 

In summary, we believe the comparison site selection process produced a set of comparison 
districts whose characteristics matched those of the pilot districts fairly closely. Although the 
differences are small, the interview data indicate that some pilot-comparison district differences 

'"students in grades K-2 were excluded from this calculation, since most would not have 
been enrolled in school and thus would not have had access to the NSLP at baseline. 

"Our sample excludes students directly certified at the time of the survey. However, 
students who were not directly certified at that time (and thus are included in our sample) may 
have been directly certified in the pre-pilot period. 
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exist, which could influence the outcomes whose impacts we are measuring. Thus it is important 
to control statistically for those individual-level differences in estimating the impacts of the pilot 
projects. Accordingly, the impacts presented in Chapter IV are based on statistical models that 
have controlled carefully for pilot-comparison group differences other than the pilot intervention 
itself. Overall, we believe the comparison district samples provide a reasonable basis for 
estimating the demonstration impacts. 
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IV. IMPACTS OF THE PILOT PROJECTS 



Estimating the impacts of Up-Front Documentation and Graduated Verification as 
implemented in the pilot districts was a major goal of the evaluation. This chapter presents 
estimates of the impacts of the pilot projects on a range of outcomes. These estimates will 
address the pilot projects’ influence on deterrence, barriers, accuracy, and targeting efficiency. 

Two key questions this chapter addresses are (1) whether the pilot projects deterred 
ineligible students from becoming certified, and (2) whether the projects raised barriers to 
certification among eligible students. Accordingly, the primary analyses will involve estimating 
impacts on deterrence and barriers. However, there are other aspects of the effects of the pilots 
on certification among eligible and ineligible students, aspects related to broader measures of 
how the intervention affected the program’s overall success in providing benefits to eligible 
students and not to ineligible ones. In examining impacts on accuracy, for example, we 
measured how the pilots affected the percentage of certified students eligible for the benefits they 
are receiving. Targeting efficiency accounts for the extent to which (1) certified students are 
eligible for benefits, and (2) eligible students are certified. 

We estimated program impacts using a nonexperimental, comparison group approach as 
described in Chapter II and Volume II of this report. In particular, we carefully selected a set of 
districts to serve as comparisons to the pilot districts, and selected samples of students from both 
pilot and comparison districts. We then used survey and administrative data collected for these 
students to determine their certification status, eligibility for free or reduced-price meals, and 
other household characteristics. To determine the impact of the pilots on outcomes of interest, 
we compared mean values of the outcomes among students in pilot districts with those of 
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students in comparison districts, while controlling for other relevant household characteristics in 
a regression framework. 

One aspect of the sampling strategy affects the interpretation of results and should be noted. 
The student samples excluded students who were directly certified for free meals by the district 
and included only students who were certified by application or who were not certified. The 
rationale for excluding directly certified students was that the Up-Front Documentation and 
Graduated Verification interventions altered the procedure under which students applied for free 
or reduced-price meals and under which the districts processed the applications and conducted 
the verification. Since directly certified students were not required to complete an application to 
become certified, the pilot interventions presumably did not influence their behavior. Thus, the 
main results presented in this chapter are representative only of those students who were not 
directly certified. 

A limitation of this sampling strategy is that comparison districts were selected on the basis 
of being similar to pilot districts in terms of their overall student characteristics rather than the 
characteristics of only non-directly certified students. Thus, to the extent that pilot and 
comparison districts differed in the proportion of their students who were directly certified, their 
non-directly certified student populations could have had different characteristics even if their 
overall student populations were similar. To minimize problems caused by this limitation, we 
attempted to select comparison districts that were similar to pilot districts in their direct 
certification status.^ In addition, for selected results presented in the chapter, it was possible to 

'Overall, we were fairly successful in selecting comparison districts that were similar to 
pilot districts with respect to direct certification. Among all districts in the evaluation, 1 1 of the 
12 sets of pilot-comparison pairs matched in terms of their 2002-2003 direct certification status. 
Only the pilot district Blue Ridge, which did not use direct certification, did not match its 
comparison district, Montrose, which did use direct certification. Also, among district pairs in 
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estimate the mean value of the outcome both among non-directly certified students in a district 
and among all students in a district. Where possible, we present both sets of estimates. 

For each evaluation objective addressed in this chapter, we present our findings in two steps. 
First, we present estimates of the mean values of outcome measures among students in the pilot 
and comparison districts, without controlling for differences in the characteristics of these 
households. This gives a sense of how pilot and comparison districts differed and gives a 
context for better understanding the impact estimates. The second step involves presenting the 
impact estimates based on regressions in which preexisting differences in the characteristics of 
households in the pilot and comparison districts were controlled. 

A. THE IMPACTS OF UP-FRONT DOCUMENTATION 

Nine of the districts included in the evaluation implemented Up-Front Documentation, 
whereby students applying for free or reduced-price meals were required to provide 
documentation of their household circumstances (income/household size and/or receipt of food 
stamps, TANF, or FDPIR benefits). The impacts of Up-Front Documentation on deterrence/ 
barriers, and accuracy/targeting efficiency are shown below. 



(continued) 

which both the pilot and comparison district used direct certification, the proportion of directly 
certified students in the pilot and its comparison district tended to be similar. 

2 

Presenting estimates of mean values of outcomes among all students required us to make an 
assumption about the eligibility status of all directly certified students. Based on findings in a 
recent study that very high proportions of students approved by direct certification are eligible 
for free meals later in the school year, we assumed that all directly certified students were 
eligible for free meals (see Food and Nutrition Service 2002). 
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1. Impacts of Up-Front Documentation on Deterrence and Barriers 

a. Did Up-Front Documentation Deter Ineligible Students from Becoming Certified? 

Some students whose income made them ineligible for free or reduced-price meals were 
certified in both pilot and comparison districts. In comparison districts, for example, 4.0 percent 
of students ineligible for free or reduced-price meals were certified, and 3.9 percent of students 
ineligible for free meals alone were certified for free meals (CD_2 and CD_1 in Table IV. 1). 
Although these rates in the Up-Front Documentation comparison districts were near the low end 
of possible values, they represented a significant number of students in these districts, because 
the large majority of the students in the districts were ineligible for free or reduced-price meals. 

In the Up-Front Documentation pilot districts, the rate of certification among students 
ineligible for free or reduced-price meals was similar to the rate in comparison districts — 
3.8 percent, compared with 4.0 percent (CD_2 in Table IV. 1). Among students ineligible for 
free meals, the certification rate in pilot districts was slightly lower (3.4 percent) than the rate in 
comparison districts (3.9 percent)."^ This simple difference between pilot and comparison 
districts does not necessarily imply, however, that Up-Front Documentation caused the 
certification rate to be lower in the pilot district. To estimate the impact of Up-Front 
Documentation on certification among ineligible students, we must control for important 
differences in the characteristics of sampled households in pilot and comparison districts. 



-5 

For example, the average Up-Front Documentation comparison district had an enrollment 
of 3,351 non-directly certified students, of which 65.8 percent (or 2,205 students) were ineligible 
for free or reduced-price meals. Thus, an estimated 88 students ineligible for free or reduced- 
price meals (or 4 percent of 2,205) were certified in these districts. 

"^See Volume II for district- level estimates of certification among ineligible students. 
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TABLE IV. 1 



RATES OE CERTIEICATION AMONG INELIGIBLE STUDENTS IN UP-ERONT DOCUMENTATION 

PILOT AND COMPARISON DISTRICTS 
(Standard Errors in Parentheses) 



Percentage of Ineligible 
Students 



Certification Rate: 


Pilot 

Districts 


Comparison 

Districts 


Pree Certification Among Students with Income >130% PPL (CD_1) 


3.4 


3.9 




(0.5) 


(0.4) 


Pree/Reduced-Price Certification Among Students with Income >185% PPL 


3.8 


4.0 


(CD_2) 


(0.5) 


(0.5) 


Sample Size 

Students with Income > 130% PPL 


688 


689 


Students with Income > 185% PPL 


532 


525 



Note; Definitions of measures are provided in Table II.2. 
PPL = federal poverty level. 
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Table IV. 2 shows the estimated impaet of Up-Front Doeumentation on deterrenee, based on 
regression models deseribed in detail in Chapter II of this volume and Chapter IX of Volume II. 
The first two eolumns show regression-adjusted pereentages of ineligible students who were 
eertified for free or redueed-priee meals and are analogous to the two eolumns of results 
presented in Table IV. 1. The third eolumn shows the differenee between the regression-adjusted 
percentages in pilot versus comparison districts, the estimated impact of the pilot, and the 
standard error of this estimate. 

The estimated impacts of Up-Front Documentation on measures of deterrence are small in 
absolute value and not statistically significant. Among students ineligible for free or reduced- 
price meals, the estimated impact of the pilot on free or reduced-price certification (CD_2) is 
zero. Among students ineligible for free meals, the estimated impact of the pilot is negative, has 
a magnitude of 0.8 percentage points, and also is not statistically significant (CD_1). If this 
estimate of 0.8 percentage points were correct, it would represent a substantial reduction in the 
number of students receiving benefits they were not eligible to receive because the base to which 
it is applicable includes 80 percent of all students (that is, 80 percent of students in Up-Front 
Documentation pilot districts are not eligible for free meals). However, the estimate is not 
statistically significant, and could be due to sampling error. 

To assess the robustness of the estimated impacts of Up-Front Documentation on deterrence, 
we conducted a series of sensitivity checks that examined whether the results of our analysis 
would have differed qualitatively if (1) our procedures for imputing income had been different, 
and (2) the specification of the regression model had been different. The basic conclusions of 
our analysis did not qualitatively change in any of these alternative specifications/imputation 
procedures.^ 

^See Volume II for more details on our sensitivity checks. 
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TABLE IV.2 



ESTIMATED IMPACT OE UP-ERONT DOCUMENTATION PILOT PROJECTS ON DETERRENCE 

(Standard Errors in Parentheses) 





Regression-Adjusted Percentage of 
Ineligible Students 




Measure of Deterrence 


Pilot 


Comparison 


Impact 


Pree Certification Among Students with Income 
>130%PPL(CD_1) 


3.3 


4.1 


-0.8 

(0.78) 


Pree/Reduced-Price Certification Among Students 
with Income >185% PPL (CD_2) 


3.9 


3.9 


0.0 

(0.75) 



Note; Definitions of measures are provided in Table II.2. 

PPL = federal poverty level. 

*SignificantIy different from zero at the .05 level, two-tailed test. 
**SignificantIy different from zero at the .01 level, two-tailed test. 
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b. Did Up-Front Documentation Raise Barriers to Certification Among Eligible 

Students? 

Up-Front Documentation could have created barriers to certification of eligible students 
through at least four mechanisms: (1) the household decided not to apply because they did not 
want to provide documents to the school district; (2) the household wanted to apply but did not 
have the necessary documents; (3) the household submitted an application without 
documentation or with documentation that the school district judged to be incomplete, and then 
never submitted complete documentation; or (4) the SFA may have had an increased probability 
of making an administrative error resulting in the denial of free or reduced-price meal benefits to 
an income-eligible household because of the increased complexity of the administrative process. 
This may not be an exhaustive list of ways in which barriers could be created.^ 

Certification for free or reduced-price meals among eligible students in the Up-Front 
Documentation evaluation districts was far from universal. Excluding directly certified students, 
in comparison districts, only about half (53.9 percent) of students eligible for free meals were 
certified for them (CB_la) and 62.4 percent of those eligible for free meals were approved for 
either free or reduced-price meals (CB_2a) (Table IV. 3). Similarly, among students eligible for 
free or reduced-price meals but not directly certified, 50.5 percent were certified for free or 
reduced-price meals (Cb_3a). Among FS/TANF recipients not directly certified, 69.7 percent 
were certified free by application (CB_4a). 

Certification rates among all students eligible for benefits (including those directly certified) 
were somewhat higher. For example, 58.9 percent of all students eligible for free meals in 
comparison districts were certified for them (CB_lb). The rates of certification among students 

^A future report will analyze applications in an attempt to shed light on some of the reasons. 
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TABLE IV.3 



RATES OE CERTIEICATION AMONG ELIGIBLE STUDENTS IN 
UP-ERONT DOCUMENTATION PILOT AND COMPARISON DISTRICTS 
(Standard Errors in Parentheses) 





Percentage of Eligible Students 




Pilot 


Comparison 


Certification Rate Among: 


Districts 


Districts 


Estimates Excluding Directly Certified Students 






Pree Certification Among Students with Income <= 130% PPL and Not 






Directly Certified (CB_la) 


47.3 


53.9 




(3.1) 


(2.0) 


Pree and Reduced-Price Certification Among Students with Income <= 130% 






PPL and Not Directly Certified (CB_2a) 


53.0 


62.4 




(3.3) 


(2.0) 


Pree and Reduced-Price Certification Among Students with Income <= 185% 






PPL and Not Directly Certified (CB_3a) 


42.4 


50.5 




(2.2) 


(1.6) 


Pree Certification Among PS/TANF Recipients Who Are Not Directly 






Certified (CB_4a) 


68.9 


69.7 




(6.7) 


(3.9) 


Estimates Including Directly Certified Students” 






Free Certification Among All Students with Income <= 130% FPL (CB_lb) 


55.3 


58.9 




(3.4) 


(2.8) 


Free and Reduced-Price Certification Among with Income <= 130% FPL 






(CB_2b) 


60.1 


66.6 




(3.6) 


(3.0) 


Free and Reduced-Price Certification Among All Students with Income 






<= 185% FPL (CB_3b) 


47.7 


53.7 




(2.6) 


(2.2) 


Free Certification Among All FS/TANF Recipients’’ (CB_4b) 


79.2 


78.8 




(n.a.) 


(n.a.) 


Sample Size 

Students with Income <= 130% FPL and Not Directly Certified 


262 


299 


Students with Income <= 185% FPL and Not Directly Certified 


418 


463 


Students on FS/TANF and Not Directly Certified 


103 


95 



Note; Definitions of measures are provided in Table II.2. 

“Standard errors of the estimates including directly certified students were estimated using bootstrapping methods 
whereby we selected 1,000 replicate samples with replacement from the relevant analysis sample, calculated the 
relevant estimate that included directly certified students for each replicate sample, and computed the standard 
deviation of these 1,000 bootstrapped estimates, which is reported as the standard error in the table. 

’’Standard errors could not be estimated due to small sample sizes in some sites. 

PPL = federal poverty level; n.a. = not available. 
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eligible for free meals were higher if certification was defined more broadly to include those 
certified for free or reduced-price meals. Among all students eligible for free meals in 
comparison districts (including directly certified students), 66.6 percent were certified for free or 
reduced-price meals (CB_2b). Among all those eligible for free or reduced-price meals, 53.7 
percent were certified for free or reduced-price meals (CB_3b). Certification rates were also 
higher when directly certified students are included for FS/TANF recipients because directly 
certified students comprised a larger fraction of the FS/TANF population than of the entire 
population of students eligible for free meals (CB_4b). 

Before household characteristics are controlled for, certification rates among eligible 
students in pilot districts were lower than the rates in comparison districts. Among those eligible 
for free meals, for example, 47.3 percent of those in Up-Front Documentation pilot districts were 
certified (compared with 53.9 percent in comparison districts ([CB_la]) and 53.0 percent of 
those eligible for free meals were approved for either free or reduced-price meals (compared 
with 62.4 percent in comparison districts ([CB_2a]). Including directly certified students, the 
free certification rate among eligibles (CB_lb) increased to 55.3 percent in pilot districts — closer 
to, but still below, the 58.9 percent certification rate in comparison districts.’ 

After controlling for household characteristics, the pilot is estimated to reduce the 
certification rate among some eligible students. This suggests that the Up-Front Documentation 
requirement created barriers to certification among at least some eligible students. When only 
free meal certification of students eligible for free meals is considered, the estimated impact on 
certification rates is -6.2 percentage points but not statistically significant. (See Line 1 of Table 

’See Chapter IX of Volume II for estimated certification rates among eligible students by 
district. 
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IV. 4.) However, when we consider impacts on the probability that these eligible students are 
certified for either free or reduced-price meals (CB_2a), there is a 9.3 percentage point reduction, 
and the estimate is statistically significant at the .05 level. Similarly, among those eligible for 
free or reduced-price meals (excluding students directly certified), the pilot was estimated to lead 
to a statistically significant reduction of 9.1 percentage points in the free or reduced-price 
certification rate (CB_3a in Table IV.4). These estimated effects translate into a 15 percent 
reduction in certification among students eligible for free meals and an 18 percent reduction in 
the certification rate among students eligible for free or reduced-price meals. Among FS/TANF 
recipients not directly certified, the estimated effect of the pilot on the free certification rate 
(CB_4a) was also negative but not statistically significant at the 0.05 level. 

To provide perspective on the effects of the pilot projects on barriers across the entire 
eligible population, including directly certified students, we calculated mean regression adjusted 
outcomes by district for all students, including the directly certified. These regression-adjusted 
outcomes for all eligible students were estimated as described in Chapter IX of Volume II, by 
using (1) the regression adjusted percentage of non-directly certified eligible students who were 
certified (as presented in the top panel of Table IV.4), and (2) the percentage of all eligible 
students who were directly certified. These estimates including directly certified students are 
shown in the lower panel of Table IV.4. Like the estimates including directly certified students 
shown in Table IV. 3, the certification rates of all students, including directly certified, are 
similarly higher, and the differences between them in the pilot and comparison districts — our 
estimate of the impact — are smaller than for the corresponding estimates excluding directly 
certified students. 
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TABLE IV.4 



ESTIMATED IMPACT OE UP-ERONT DOCUMENTATION PILOT PROJECTS ON BARRIERS 

(Standard Errors in Parentheses) 



Measure of Barriers 


Regression-Adjusted Percentage 
of Eligible Students 

Pilot Comparison 


Impact 


Estimates Excluding Directly Certified Students 

Pree Certification Among Students with Income <= 








130% PPL and Not Directly Certified (CB_la) 

Pree and Reduced-Price Certification Among Students 
with Income <= 130% PPL and Not Directly Certified 


47.6 


53.7 


-6.2 

(4.46) 


(CB_2a) 

Pree and Reduced-Price Certification Among Students 
with Income <= 185% PPL and Not Directly Certified 


53.1 


62.4 


-9.3* 

(4.50) 


(CB_3a) 

Pree Certification Among PS/TANP Recipients Who 


42.0 


51.1 


-9.1* 

(3.61) 


Are Not Directly Certified (CB_4a) 

Estimates Including Directly Certified Students” 

Pree Certification Among All Students with Income 


66.1 


72.6 


-6.4 

(7.57) 


<= 130% PPL (CB_lb) 

Pree and Reduced-Price Certification Among with 


55.4 


58.9 


-3.5 

(4.01) 


Income <= 130% PPL (CB_2b) 

Pree and Reduced-Price Certification Among All 


60.0 


66.7 


-6.7 

(4.17) 


Students with Income <= 185% PPL (CB_3b) 
Pree Certification Among All PS/TANP Recipients’’ 


47.1 


54.3 


-7.2* 

(2.96) 


(CB_4b) 


74.5 


81.3 


-6.7 

(NA) 



Note: Definitions of measures are provided in Table II.2. 

“Standard errors of the estimates including directly certified students were estimated using bootstrapping 
methods whereby we selected 1,000 replicate samples with replacement from the relevant analysis sample, 
calculated the relevant estimate that included directly certified students for each replicate sample, and 
computed the standard deviation of these 1,000 bootstrapped estimates, which is reported as the standard error 
in the table. 

’’Standard errors could not be estimated due to small sample sizes in some sites. 

PPL = federal poverty level; NA = not available. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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2. Impacts of Up-Front Documentation on Program Integrity: Summary Measures 

An important objective of the NSLP’s system for determining eligibility for free or reduced- 
price meals is to ensure that benefits go to students who are eligible for them (and who want to 
receive them) and do not go to students who are ineligible. Districts with the highest levels of 
deterrence among ineligible students and the fewest barriers to certification among eligible 
students will have the greatest success in meeting this objective. So far, this chapter has 
presented the results of analysis that examines deterrence and barriers separately. We now 
present estimates of the influence of Up-Front Documentation on two sets of program success 
measures that summarize the effects on deterrence and barriers: (1) certification accuracy, and 
(2) targeting efficiency. Certification accuracy is measured by the percentage of students 
certified for free or reduced-price meals who are eligible for the level of benefits they are 
receiving. Targeting efficiency measures the extent to which eligible students receive benefits 
and ineligible students do not. 

a. How Did Up-Front Documentation Affect Certification Accuracy? 

Most certified students in the Up-Front Documentation evaluation districts were eligible for 
the benefits they were receiving. Excluding directly certified students, 77.5 percent of students 
certified for free meals in comparison districts were eligible for these benefits (CA_la), and 86.5 
percent of those certified for free or reduced-price meals had incomes less than 185 percent of 
poverty (CA_4a) (Table IV. 5).^ We also calculated accuracy for free and reduced-price meals 

o 

Among the group with incomes not over 185 percent of the federal poverty level, the 
measure (CA_4a) counts as accurately certified children who are eligible for reduced-price meals 
but certified for free meals. While these children are not correctly certified, the amount of 
erroneous reimbursement that FNS is at risk of making is smaller than the erroneous 
reimbursement for a meal received by children approved for free meals who are eligible for free 
or reduced-price meals. The free and reduced-price accuracy measure (CA_2a) and not 
overcertified measures (CA_3a) count children eligible for reduced-price meals who are certified 
for free meals as inaccurately certified. 
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TABLE IV.5 



CERTIFICATION ACCURACY RATES AMONG CERTIFIED STUDENTS IN 
UP-FRONT DOCUMENTATION PILOT AND COMPARISON DISTRICTS 
(Standard Errors in Parentheses) 



Certification Accuracy Rate 


Percentage of Certified Students 

Pilot Comparison 

Districts Districts 


Estimates Excluding Directly Certified Students 


Free Certification Accuracy Among Free Approved, Non-Directly Certified 


79.6 


77.5 


Students (CA_la) 


(2.8) 


(2.7) 


Free and Reduced-Price Certification Accuracy Among Free and Reduced-Price 


71.2 


68.8 


Approved, non-Directly Certified Students (CA_2a) 


(2.6) 


(2.6) 


Not Overcertified Among Free and Reduced-Price Approved, Non-Directly 


78.3 


77.8 


Certified Students (CA_3a) 


(2.4) 


(2.5) 


Not Over 185% FPL Among Free and Reduced-Price Approved, Non-Directly 


85.6 


86.5 


Certified Students'" (CA_4a) 


(1.9) 


(1.9) 


Estimates Including Directly Certified Students'’ 


Free Certification Accuracy Among All Free Approved Students (CA_lb) 


84.7 


81.4 




(2.1) 


(2.5) 


Free and Reduced-Price Certification Accuracy Among All Free and Reduced- 


76.8 


72.4 


Price Approved Students (CA_2b) 


(2.3) 


(2.3) 


Not Overcertified Among All Free and Reduced-Price Approved Students 


82.5 


80.4 


(CA_3b) 


(2.0) 


(2.1) 


Not Over 185% FPL Among All Free and Reduced-Price Approved Students'" 


88.6 


87.9 


(CA_4b) 


(1.6) 


(1.7) 


Sample Size 

Students Certified for Free Meals and Not Directly Certified 


204 


222 


Students Certified for Reduced-Price Meals and Not Directly Certified 


319 


376 



Note: Definitions of measures are provided in Table II.2. 

'‘Students in group C in Table II. 1 are eligible for reduced-price meals but approved for free meals. Accordingly, these 
students are excluded from the numerator of the Free and Reduced-Price Certification Accuracy measure (CA_2a) (that 
is, not counted as correctly certified). However, they are included in the numerator of the “not over 185% FPL” accuracy 
measure (CA_4a) (that is, counted as correctly certified). The rationale for counting these cases as correct in measure 
CA_4a is that the cost to the federal government of this error is lower than the costs of errors in approving students 
income exceeding 185% FPL. When children eligible for reduced-price meals but approved for free meals get a NSLP- 
reimbursable meal, the amount of the erroneous payments that FNS is at risk of making to school districts is limited to 40 
cents per meal — the difference between a reduced-price reimbursement and a free reimbursement. It is much smaller 
than the amount of erroneous payments FNS is at risk of making to school districts reimbursed for meals received by 
children approved for free meals who are eligible for neither free nor reduced-price meals. 

'’Standard errors of the estimates including directly certified students were estimated using bootstrapping methods whereby 
we selected 1,000 replicate samples with replacement from the relevant analysis sample, calculated the relevant estimate 
that included directly certified students for each replicate sample, and computed the standard deviation of these 1,000 
bootstrapped estimates, which is reported as the standard error in the table. 
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using a precise definition according to which only those with incomes less than 130 percent of 
poverty and certified free plus those with incomes between 131 and 185 percent of poverty and 
certified reduced-price are counted as accurately certified. Under this definition (CA_2a), 68.8 
percent of certified cases were accurately certified. Finally, using a definition which counts as 
accurate any student who is not certified for a higher level of benefits than the student is eligible 
for (CA_3a), 77.8 percent of non-directly certified students are accurately certified. Including 
directly certified students increased the accuracy rate slightly for the measures shown in the 
table. For example, free meal accuracy increased from 77.5 to 81.4 percent. 

Accuracy rates in Up-Front Documentation pilot districts were roughly the same as rates in 
comparison districts. Table IV. 5 shows the accuracy rate for free meals (excluding directly 
certified students) (CA_la) to have been slightly higher in pilot than comparison districts (79.6 
versus 77.5 percent), but the percentage certified for free or reduced-price meals who had 
incomes less than 185 percent of poverty (CA_4a) to have been slightly lower in pilot districts 
(85.6 versus 86.5 percent). 

To estimate the impact of the pilot on accuracy, we used the results of the models that 
estimated the impacts of the pilot on deterrence and barriers (along with information on the 
percentage of eligible and ineligible students in each district), as described in Chapter IX of 
Volume II. This methodology highlights the fact that accuracy rates in the evaluation districts 
were influenced by the extent to which the districts deterred ineligible households from 
becoming certified and, conversely, lowered barriers and promoted certification among eligible 
students. Deterring ineligible students from becoming certified improved accuracy, for obvious 
reasons. However, lessening barriers to certification among ineligible households also improved 
accuracy, since each additional eligible student who became certified, all else equal, pushed the 
accuracy rate — the proportion of all certified students who were eligible — closer to one. 
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Correspondingly, greater barriers to certification among eligible students reduced accuracy, since 
having fewer certified students who were eligible implied that districts would have 
proportionally more who were ineligible, all else equal. 

Overall, Up-Front Documentation did not have a statistically significant effect on the 
accuracy rate. The estimated impact of the pilot on the accuracy rate among those certified for 
free meals (CA_la) was small and positive, while the estimated impact on the percentage of 
those certified for free or reduced-price meals who had incomes of no more than 185 percent of 
poverty (CA_4a) was small and negative (Table IV. 6). In each case, however, the estimate was 
not statistically significant at the 0.05 level and may have been due to sampling variability. 

b. How Did Up-Front Documentation Affect Targeting Efficiency? 

The targeting efficiency rate is defined as the percentage of a district’s students whose 
eligibility and certification statuses were consistent with each other — they were either eligible 
and certified or not eligible and not certified. Targeting efficiency is lower to the extent that a 
district has either students who are eligible but not certified (because they faced barriers) or are 
not eligible but certified (because deterrence failed). Again, the impact of Up-Front 
Documentation on targeting efficiency depended on its effects on both deterrence and barriers. 

We present three targeting efficiency measures. These measures correspond to the three 
measures for accuracy among students certified for free or reduced-price meals presented in 
Table IV. 5. For example, the first targeting efficiency measure excluding directly certified 
students (CTE_la) counts as being correctly targeted: (1) students certified for free meals with 
incomes less than 130 percent of poverty, (2) students certified for reduced-price meals with 
incomes 131-185 percent of poverty, and (3) students not certified for free or reduced-price 



72 




TABLE IV.6 



ESTIMATED IMPACT OE UP-ERONT DOCUMENTATION PILOT PROJECTS ON 

CERTIEICATION ACCURACY 
(Standard Errors in Parentheses) 



Measure of Certification Accuracy 


Regression- Adjusted Percentage of 
Certified Students 

Pilot Comparison 


Impact 


Tree Certification Accuracy Among Tree 
Approved, Non-Directly Certified Students 








(CA_la) 

Not Over 185% PPL Among Pree and 
Reduced-Price Approved, Non-Directly 


78.2 


75.9 


2.3 

(5.2) 


Certified Students (CA_4a) 


82.7 


85.6 


-2.9 

(3.8) 



Notes: Definitions of measures are provided in Table II.2. Standard errors of the impact estimates were estimated 
using bootstrapping methods, whereby 1,000 subsamples were selected with replacement from the relevant 
analysis sample, the impact estimates were computed with each subsample, and the standard deviation of 
these 1,000 impact estimates was computed, and reported as the standard error. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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meals with incomes above 185 percent of poverty. The targeting efficiency rate is defined as the 
number of students correctly classified as a percentage of all students not directly certified. 

More than three-quarters of students in Up-Front Documentation pilot and comparison 
districts were “efficiently targeted.” In comparison districts, for example, using the first 
definition of targeting efficiency (CTE_la), 77.4 percent of students had a certification status for 
free or reduced-price meals consistent with their eligibility status (Table IV. 7). Using the third 
measure of accuracy and targeting efficiency (CTE_3a), 80.8 percent of non-directly certified 
students had a certification status consistent with their eligibility status. Targeting efficiency was 
affected by inclusion of directly certified students only to a small degree because directly 
certified students are a relatively small fraction (about 3 percent on average across the districts) 
of all students in the Up-Eront Documentation districts (CTE_lb, CTE_2b, and CTE_3b in Table 
IV. 7). Einally, across all measures, targeting efficiency rates in pilot districts were nearly 
identical to the efficiency rates in the comparison districts. 

The estimated impact of Up-Eront Documentation on targeting efficiency was small and not 
statistically significant. Controlling for household characteristics, the regression-adjusted 
targeting efficiency rates in pilot and comparison districts (CTE_3a) were close to one another, 
and the small negative impact could have been due to sampling variability (Table IV. 8). 

B. THE IMPACTS OF GRADUATED VERIFICATION 

The second approach taken by districts participating in the demonstration and included in 
the evaluation was to increase the scale of their verification efforts, if initial verification efforts 
resulted in large proportions of verified households whose benefits were reduced or terminated. 
Three districts included in the evaluation implemented the Graduated Verification pilot program. 
As with Up-Front Documentation, the purpose of Graduated Verification was to deter ineligible 
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TABLE IV.7 



RATES OE TARGETING EEEICIENCY AMONG STUDENTS IN UP-ERONT DOCUMENTATION 

PILOT AND COMPARISON DISTRICTS 
(Standard Errors in Parentheses) 



Targeting Efficiency Rate 


Percentage of Students 

Pilot Comparison 

Districts Districts 


Estimates Excluding Directly Certified Students 


Targeting Efficiency Measure 1 Among Non-Directly Certified Students 


77.6 


77.4 


(CTE_la) 


(1.5) 


(1.4) 


Targeting Efficiency Measure 2 Among Non-Directly Certified Students 


78.7 


79.1 


(CTE_2a) 


(1.5) 


(1.4) 


Targeting Efficiency Measure 3 Among Non-Directly Certified Students 


80.2 


80.8 


(CTE_3a) 


(1.4) 


(1.4) 


Estimates Including Directly Certified Students^ 


Targeting Efficiency Measure 1 Among All Students (CTE_lb) 


78.3 


77.9 




(1.4) 


(1.3) 


Targeting Efficiency Measure 2 Among All Students (CTE_2b) 


79.3 


79.6 




(1.4) 


(1.3) 


Targeting Efficiency Measure 3 Among All Students (CTE_3b) 


80.8 


81.3 


Sample Size 


(1.4) 


(1.3) 


All Students 


950 


988 



Notes: Definitions of measures are provided in Table II.2. Efficient targeting is defined as being correctly assigned 
to either the certified or the noncertified group given household income. It is calculated as the percentage of 
students who are either (1) eligible for benefits and certified, or (2) not eligible for benefits and not certified. 

“Standard errors of the estimates including directly certified students were estimated using bootstrapping methods 
whereby we selected 1,000 replicate samples with replacement from the relevant analysis sample, calculated the 
relevant estimate that included directly certified students for each replicate sample, and computed the standard 
deviation of these 1,000 bootstrapped estimates, which is reported as the standard error in the table. 
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TABLE IV.8 



ESTIMATED IMPACT OE UP-ERONT DOCUMENTATION PILOT PROJECTS ON TARGETING 

EEEICIENCY 

(Standard Errors in Parentheses) 



Regression-Adjusted Percentage of All 
Students 



Measure of Targeting Efficiency 



Pilot Comparison Impact 



Targeting Efficiency Measure 3 Among Non- 

Directly Certified Students (CTE_3a) 79.4 81.2 -1.8 

( 1 - 8 ) 



Notes: Definitions of measures are provided in Table II.2. Standard errors of the impact estimates were estimated 
using bootstrapping methods, whereby 1,000 subsamples were selected with replacement from the relevant 
analysis sample, the impact estimates were computed with each subsample, and the variance of the estimates 
in these repeated calculations was computed. 

Although we were unable to estimate impacts on targeting efficiency measure 1 among non-directly certified 
students for this report, we plan to estimate these impacts and include them in a supplemental appendix to 
this report to be completed later. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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students from becoming and remaining certified without reducing certification levels among 
eligible students. 

1. Impacts of Graduated Verification on Deterrence and Barriers 

a. Did Graduated Verification Deter Ineligible Students from Becoming Certified? 

About 1 in 10 students (9.7 percent) ineligible for free or reduced-price meals in Graduated 
Verification comparison districts were certified, as were 8.6 percent of those ineligible for free 
meals (CD_2 and CD_1 in Table IV. 9). Certification rates among ineligible students in pilot 
districts were somewhat lower than those in comparison districts. Among those ineligible for free 
or reduced-price meals in pilot districts (CD_2), only 4.8 percent were certified, about half the 
rate in comparison districts. The pilot-comparison difference in certification among students 
ineligible for free meals (CD_1) was not as large, with 7.4 percent of those in pilot districts 
certified for free meals. ^ 

After controlling for household characteristics, the pilot-comparison differences in 
deterrence were smaller and were not statistically significant (Table IV. 10). Among students 
ineligible for free or reduced-price meals (CD_2), the estimated impact of the pilot on the 
free/reduced-price certification rate was negative and relatively large (-2.5 percentage points, a 
reduction of nearly 30 percent) but not statistically significant.^'^ Among students ineligible for 
free meals (CD_1), the estimated impact of the pilot was zero.^^ 

^See Chapter IX of Volume II for district-level certification rates among ineligible students. 

"’The 95 percent confidence interval of this estimate was -5.93 to -1-0.93 percentage points. 

"We conducted a set of sensitivity checks on the findings regarding the estimated impacts 
of Graduated Verification similar to the set we conducted on the estimated impacts of Up-Front 
Verification. Again, we found that the conclusions from our primary analysis — presented in this 
chapter — were not substantially altered. 
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TABLE IV.9 



RATES OE CERTIEICATION AMONG INELIGIBLE STUDENTS IN GRADUATED VERIEICATION 

PILOT AND COMPARISON DISTRICTS 
(Standard Errors in Parentheses) 





Percentage of Ineligible 
Students 


Certification Rate: 


Pilot 

Districts 


Comparison 

Districts 


Pree Certification Among Students with Income >130% PPL (CD_1) 


7.4 


8.6 




(0.9) 


(1.1) 


Pree/Reduced-Price Certification Among Students with Income >185% PPL 


4.8 


9.7 


(CD_2) 


(0.7) 


(2.1) 


Sample Size 

Students with Income > 130% PPL 


310 


319 


Students with Income > 185% PPL 


202 


221 



Note; Definitions of measnres are provided in Table II.2. 
PPL = federal poverty level. 
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TABLE IV. 10 



ESTIMATED IMPACT OE GRADUATED VERIEICATION PILOT 
PROJECTS ON DETERRENCE 
(Standard Errors in Parentheses) 





Regression-Adjusted Percentage of 
Ineligible Students 




Measure of Deterrence 


Pilot 


Comparison 


Impact 


free Certification Among Students with 
Income >130% PPL (CD_1) 


8.0 


8.0 


0.0 

(2.25) 


Pree/Reduced-Price Certification Among 
Students with Income >185% PPL (CD_2) 


6.1 


8.6 


-2.5 

(1.75) 



Note; Definitions of measnres are provided in Table II.2. 

PPL = federal poverty level. 

*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 
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Thus, the estimates provided no evidence that Graduated Verification had a large impact on 
certification among ineligible students. However, the limited statistical power of the analysis 
leaves more uncertainty about whether the pilot had a small or moderate deterrent effect. 

b. Did Graduated Verification Raise Barriers to Certification Among Eligible Students? 

Graduated Verification could have raised barriers to certification among eligible students in 
two ways. First, because students certified for free or reduced-price meals were more likely to 
be subjected to the verification process in Graduated Verification districts, any households not 
wishing to go through this process might have been discouraged from applying for free or 
reduced-price meals in the first place. Second, households that were subjected to verification in 
the previous year and who had their benefits reduced or terminated were required by the 
Graduated Verification procedures to submit documentation the next time they applied for 
benefits, if this happened within a year of the time their benefits were reduced or terminated. 
Although the pilot districts did not always implement this requirement consistently, either this 
requirement or the stigmatizing effects of having their benefits in the previous year cut could 
have discouraged households from subsequently applying for benefits. 

In Graduated Verification comparison districts, 69.1 percent of non-directly certified 
students eligible for free meals were certified for these benefits, and 81.1 percent of those 
eligible for free meals were approved for free or reduced-price meals (CB_la and CB_2a, in 
Table IV. 11). Among those eligible for free or reduced-price meals, 72.2 percent were certified 
for free or reduced-price meals (CB_3a). Among FS/TANF recipients not directly certified, 72.4 
percent were certified (CB_4a). Including directly certified students, 79.2 percent of comparison 

I 9 

A forthcoming report on the evaluation will describe implementation of the pilot 
procedures. 
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TABLEIV.il 



RATES OE CERTIEICATION AMONG ELIGIBLE STUDENTS IN GRADUATED VERIEICATION 

PILOT AND COMPARISON DISTRICTS 
(Standard Errors in Parentheses) 





Percentage of Eligible Students 




Pilot 


Comparison 


Certification Rate Among: 


Districts 


Districts 


Estimates Excluding Directly Certified Students 






Pree Certification Among Students with Income <= 130% PPL and Not 


55.8 


69.1 


Directly Certified (CB_la) 


(4.3) 


(4.0) 


Pree and Reduced-Price Certification Among Students with Income <= 130% 


62.2 


81.1 


PPL and Not Directly Certified (CB_2a) 


(4.6) 


(4.2) 


Pree and Reduced-Price Certification Among Students with Income <= 185% 


60.4 


72.2 


PPL and Not Directly Certified (CB_3a) 


(3.3) 


(3.0) 


Pree Certification Among PS/TANP Recipients Who Are Not Directly 


69.2 


72.4 


Certified (CB_4a) 


(11.2) 


(9.5) 


Estimates Including Directly Certified Students^ 






Pree Certification Among All Students with Income <= 130% PPL (CB_lb) 


72.0 


79.2 




(3.1) 


(3.1) 


Pree and Reduced-Price Certification Among with Income <= 130% PPL 


76.2 


87.0 


(CB_2b) 


(3.1) 


(2.9) 


Pree and Reduced-Price Certification Among All Students with Income 


70.3 


78.5 


<= 185% PPL (CB_3b) 


(2.8) 


(3.1) 


Pree Certification Among All PS/TANP Recipients’’ (CB_4b) 


87.2 


88.0 




(3.5) 


(4.1) 


Sample Size 

Students with Income <= 130% PPL and Not Directly Certified 


220 


216 


Students with Income <= 185% PPL and Not Directly Certified 


328 


314 


Students on PS/TANP and Not Directly Certified 


107 


90 



Note; Definitions of measnres are provided in Table 11.2. 

“Standard errors of the estimates including directly certified students were estimated using bootstrapping methods 
whereby we selected 1,000 replicate samples with replacement from the relevant analysis sample, calculated the 
relevant estimate that included directly certified students for each replicate sample, and computed the standard 
deviation of these 1,000 bootstrapped estimates, which is reported as the standard error in the table. 

PPL = federal poverty level. 
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district students eligible for free meals (CB_lb) were certified for these benefits, while 78.5 
percent of those eligible for free or reduced-price meals (CB_3b) were certified for either free or 
reduced-price meals. 

Certification rates among eligible students in pilot districts were substantially lower than the 
rates in comparison districts. For example, 55.8 percent of eligible students in Graduated 
Verification pilot districts were certified for free meals, compared with 69.1 percent among 
eligible comparison group students (CB_la). Including directly certified students reduced this 
difference somewhat, but not did eliminate it (CB_lb). Similar differences existed in the rates of 
certification for free or reduced-price meals among all eligible students in pilot versus 
comparison districts. 

The differences between the pilot and comparison districts in the certification rates of 
students eligible for free meals persisted after controlling for household characteristics. Among 
students eligible for free meals and not directly certified, the estimated impact on the rate of free 
certification was -13.3 percentage points, and the estimated impact on the rate of free or 
reduced-price certification was -15.6 percentage points (CB_la and CB_2a in Table IV. 12). 
Both estimates were statistically significant. The estimated impact on the certification rate 
among non-directly certified students eligible for free or reduced-price meals (CB_3a) was -9.1 
percentage points. This point estimate was not statistically significant at the .05 confidence 
level. Finally, the estimated impact on the certification rate among FS/TANF recipients not 
directly certified (CB_4a) was -12.7, but, again, not statistically significant. 

The impact estimates for each certification measure for all eligible students, including those 
directly certified, was smaller than the corresponding estimate when directly certified students 
are included (CB_lb - CB_4b in Table IV. 12). Patterns of statistical significance are likewise 
similar. 
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TABLE IV. 12 



ESTIMATED IMPACT OE GRADUATED VERIEICATION PILOT PROJECTS ON BARRIERS 

(Standard Errors in Parentheses) 



Regression- Adjusted 
Percentage of Eligible Students 



Measure of Barriers 


Pilot 


Comparison 


Impact 


Estimates Excluding Directly Certified Students 


Eree Certification Among Students with Income <= 130% 
EPL and Not Directly Certified (CB_la) 


56.0 


69.3 


-13.3* 

(6.14) 


Eree and Reduced-Price Certification Among Students 
with Income <= 130% PPL and Not Directly Certified 
(CB_2a) 


64.2 


79.9 


-15.6** 

(6.16) 


Eree and Reduced-Price Certification Among Students 
with Income <= 185% PPL and Not Directly Certified 
(CB_3a) 


62.0 


71.1 


-9.1 

(5.33) 


Eree Certification Among PS/TANP Recipients Who Are 
Not Directly Certified (CB_4a) 


64.8 


77.5 


-12.7 

(9.97) 


Estimates Including Directly Certified Students^ 


Eree Certification Among All Students with Income 
<= 130% PPL (CB_lb) 


72.2 


79.2 


-7.0 

(3.70) 


Eree and Reduced-Price Certification Among with Income 
<= 130% PPL (CB_2b) 


77.4 


86.2 


-8.8* 

(3.56) 


Eree and Reduced-Price Certification Among All Students 
with Income <= 185% PPL (CB_3b) 


71.2 


77.6 


-6.4 

(3.73) 


Eree Certification Among All PS/TANP Recipients’’ 
(CB_4b) 


85.8 


89.8 


-4.0 

(4.20) 



Note; Definitions of measures are provided in Table II.2. 

“Standard errors of the estimates including directly certified students were estimated using bootstrapping methods 
whereby we selected 1,000 replicate samples with replacement from the relevant analysis sample, calculated the 
relevant estimate that included directly certified students for each replicate sample, and computed the standard 
deviation of these 1,000 bootstrapped estimates, which is reported as the standard error in the table. 

EPL = federal poverty level. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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2. Impacts of Graduated Verification on Program Integrity: Summary Measures 
a. How Did Graduated Verification Affect Certification Accuracy? 

As in Up-Front Documentation districts, most certified students in Graduated Verification 
districts were eligible for the level of benefits they were receiving at the time of the survey. 
Excluding directly certified students, 73.0 percent of those certified for free meals in comparison 
districts had incomes of no more than 130 percent of the federal poverty level, which made them 
eligible for these benefits (CA_la in Table IV. 13). Including directly certified students, this 
accuracy rate for free meals was 81.9 percent (CA_lb). Among students certified for free or 
reduced-price meals in comparison districts, 85.3 percent had incomes of no more than 185 
percent of poverty with directly certified students excluded (CA_4a), and 89.0 percent had 
incomes at this level with directly certified students included (CA_4b). Using the precisely 
correct definition according to which only those with incomes less than 130 percent of poverty 
and certified free plus those with incomes between 131 and 185 percent of poverty and certified 
reduced-price were counted as accurately certified (CA_2a), 64.9 percent of non-directly 
certified cases are accurately certified. Finally, using a definition which counts as accurate any 
student who was not certified for a higher level of benefits than the student was eligible for 
(CA_3a), 72.4 percent of non-directly certified students were accurately certified. Accuracy 
rates for free meals in Graduated Verification pilot districts among those certified by application 
were slightly lower than rates in comparison districts (71.6 compared to 73.0 percent). The 

1 T 

Among the group with incomes not over 185 percent of the federal poverty level, this 
measure counts as correctly certified children who are eligible for reduced-price meals but 
certified for free meals. While these children are not correctly certified, the amount of erroneous 
payments that FNS is at risk of making is smaller than the erroneous reimbursement for a meal 
received by children approved for free meals who are eligible for free or reduced-price meals. 
The free and reduced-rice accuracy (measure CA_2a) and not overcertified measures (CA_3a) 
count children eligible for reduced-price meals who are certified for free meals as inaccurately 
certified. 
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TABLE IV. 13 



CERTIFICATION ACCURACY RATES AMONG CERTIFIED STUDENTS IN 
GRADUATED VERIFICATION PILOT AND COMPARISON DISTRICTS 
(Standard Errors in Parentheses) 



Accuracy Rate 


Percentage of Certified Students 

Pilot Comparison 

Districts Districts 


Estimates Excluding Directly Certified Students 


Free Certification Accuracy Among Free Approved, Non-Directly Certified 


71.6 


73.0 


Students (CA_la) 


(4.4) 


(3.3) 


Free and Reduced-Price Certification Accuracy Among Free and Reduced-Price 


69.1 


64.9 


Approved, non-Directly Certified Students (CA_2a) 


(3.7) 


(3.4) 


Not Overcertified Among Free and Reduced-Price Approved, Non-Directly 


75.2 


72.4 


Certified Students (CA_3a) 


(3.3) 


(3.3) 


Not Over 185% FPL Among Free and Reduced-Price Approved, Non-Directly 


90.8 


85.3 


Certified Students'" (CA_4a) 


(1.9) 


(2.9) 


Estimates Including Directly Certified Students'’ 


Free Certification Accuracy Among All Free Approved Students (CA_lb) 


84.4 


81.9 




(2.1) 


(2.1) 


Free and Reduced-Price Certification Accuracy Among All Free and Reduced- 


80.0 


73.7 


Price Approved Students (CA_2b) 


(2.1) 


(2.5) 


Not Overcertified Among All Free and Reduced-Price Approved Students 


84.1 


79.3 


(CA_3b) 


(1-9) 


(2.4) 


Not Over 185% FPL Among All Free and Reduced-Price Approved Students'" 


94.1 


89.0 


(CA_4b) 


(1.2) 


(2.1) 


Sample Size 

Students Certified for Free Meals and Not Directly Certified 


221 


229 


Students Certified for Reduced-Price Meals and Not Directly Certified 


287 


314 



Note: Definitions of measures are provided in Table II.2. 

'‘Students in group C in Table II. 1 are eligible for reduced-price meals but approved for free meals. Accordingly, these 
students are excluded from the numerator of the Free and Reduced-Price Certification Accuracy measure (CA_2a) (that 
is, not counted as correctly certified). However, they are included in the numerator of the “not over 185% FPL” accuracy 
measure (CA_4a) (that is, counted as correctly certified). The rationale for counting those cases as correct in measure 
CA_4a is that the cost to the federal government of this error is lower than the costs of errors in approving students with 
incomes exceeding 185% FPL. When children eligible for reduced-price meals but approved for free meals get a NSLP- 
reimbursable meal, the amount of the erroneous payments that FNS is at risk of making to school districts is limited to 40 
cents per meal — the difference between a reduced-price reimbursement and a free reimbursement. It is much smaller 
than the amount of erroneous payments FNS is at risk of making to school districts reimbursed for meals received by 
children approved for free meals who are eligible for neither free nor reduced-price meals. 

'’Standard errors of the estimates including directly certified students were estimated using bootstrapping methods whereby 
we selected 1,000 replicate samples with replacement from the relevant analysis sample, calculated the relevant estimate 
that included directly certified students for each replicate sample, and computed the standard deviation of these 1,000 
bootstrapped estimates, which is reported as the standard error in the table. 

FPL = federal poverty level. 
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percentage of students certified for free or reduced-price meals who were eligible was higher in 
Graduated Verification pilot districts than in comparison districts for all three of the accuracy 
measures examined when directly certified students were excluded. When directly certified 
students were included, all four accuracy measures were higher in the pilot sites than in the 
comparison sites. 

The estimated impacts of Graduated Verification were mixed, and not statistically 
significant. The estimated impact of the pilot on the accuracy rate for free meals was negative 
and not statistically significant, while the impact on the percentage of cases certified for free or 
reduced-price meals who were in households with incomes below 185 percent of poverty was 
positive and not statistically significant (CA_la and CA_4a in Table IV. 14). This finding of 
inconsistent estimated effects on accuracy for free and for free or reduced-price meals and large 
standard errors of the estimates is similar to the pattern of findings of Up-Front Documentation 
on accuracy rates, presented in the previous section. 

b. How Did Graduated Verification Affect Targeting Efficiency? 

More than three-quarters of students in Graduated Verification pilot and comparison districts 
were “efficiently targeted.” In comparison districts, for example, using the first definition of 
targeting efficiency, 75.6 percent of students had a certification status for free or reduced-price 
meals consistent with their eligibility status (CTE_la in Table IV. 15). Using the broadest third 
measure of targeting efficiency (CTE_3a), 82.6 percent of non-directly certified students had a 
certification status consistent with their eligibility status. Including directly certified students in 
the targeting efficiency measure increased the percentage correctly targeted by two to three 
percentage points. Einally, targeting efficiency rates in pilot districts were nearly identical to the 
efficiency rates in the comparison districts. 
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TABLE IV. 14 



ESTIMATED IMPACT OE GRADUATED VERIEICATION PILOT PROJECTS ON 
CERTIEICATION ACCURACY 
(Standard Errors in Parentheses) 





Regression- Adjusted Percentage of 
Certified Students 




Measure of Accuracy 


Pilot 


Comparison 


Impact 


free Certification Accuracy Among free Approved, 
Non-Directly Certified Students (CA_la) 


70.5 


74.3 


-3.7 

(5.2) 


Not Over 185% PPL Among Pree and Reduced-Price 
Approved, Non-Directly Certified Students 
(CA_4a) 


88.9 


86.4 


2.6 

(3.7) 



Notes: Definitions of measures are provided in Table II.2. Standard errors of the impact estimates were estimated 
using bootstrapping methods, whereby 1,000 replicate samples were selected with replacement from the 
relevant analysis sample, the impact estimates were computed with each subsample, and the standard 
deviation of these 1,000 impact estimates was computed and reported as the standard error. 

PPL = federal poverty level. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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TABLE IV. 15 



RATES OE TARGETING EEEICIENCY AMONG STUDENTS IN GRADUATED VERIEICATION 

PILOT AND COMPARISON DISTRICTS 
(Standard Errors in Parentheses) 





Percentage of Students 




Pilot 


Comparison 


Targeting Efficiency Rate 


Districts 


Districts 


Estimates Excluding Directly Certified Students 






Targeting Efficiency Measure 1 Among Non-Directly Certified Students 


74.3 


75.6 


(CTE_la) 


(2.2) 


(2.4) 


Targeting Efficiency Measure 2 Among Non-Directly Certified Students 


76.6 


78.3 


(CTE_2a) 


(2.2) 


(2.4) 


Targeting Efficiency Measure 3 Among Non-Directly Certified Students 


80.4 


82.6 


(CTE_3a) 


(2.1) 


(2.4) 


Estimates Including Directly Certified Students^ 






Targeting Efficiency Measure 1 Among All Students (CTE_lb) 


77.4 


78.6 




(1.9) 


(2.0) 


Targeting Efficiency Measure 2 Among All Students (CTE_2b) 


79.2 


80.9 




(1.8) 


(2.0) 


Targeting Efficiency Measure 3 Among All Students (CTE_3b) 


82.7 


84.8 




(1.8) 


(2.0) 


Sample Size 






All Students 


530 


535 



Notes: Definitions of measures are provided in Table II.2. Efficient targeting is defined as being correctly assigned 
to either the certified or the noncertified group given household income. It is calculated as the percentage of 
students who are either (1) eligible for benefits and certified, or (2) not eligible for benefits and not certified. 

“Standard errors of the estimates including directly certified students were estimated using bootstrapping methods 
whereby we selected 1,000 replicate samples with replacement from the relevant analysis sample, calculated the 
relevant estimate that included directly certified students for each replicate sample, and computed the standard 
deviation of these 1,000 bootstrapped estimates, which is reported as the standard error in the table. 
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After controlling for household characteristics, the estimated impact of Graduated 
Verification on targeting efficiency was small (-3.2 percentage points) and not statistically 
significant at the .05 level (Table IV. 16). 

C. INCOME LEVELS OF INELIGIBLE STUDENTS WHO ARE CERTIFIED 

The findings on deterrence indicate that approximately 4 percent of ineligible households 
were certified for free or reduced-price meals in Up-Front Documentation comparison districts 
(Table IV. 1) and that 9 to 10 percent of ineligible households were certified in Graduated 
Verification comparison districts (Table IV. 9). This section presents data on the income 
distribution of these ineligible certified families. 

Table IV. 17 provides the relevant data for Up-Front Documentation and Graduated 
Verification pilot and comparison districts. The table shows the percentages of the total 
ineligible population who are certified (repeated from Table IV. 1 and Table IV.9). The income 
distribution of the ineligible households is shown below each estimate of the percentage in the 
total eligible population. For example, in the Up-Front Documentation comparison districts, 3.9 
percent of students not eligible for free meals (households with income above 130 percent of the 
federal poverty level) were certified for free meals. Among those ineligible certified households 
in our sample in the Up-Front Documentation comparison sites, 66.4 percent had incomes 
between 131 and 185 percent of the federal poverty level, 25.3 percent had incomes between 186 
and 250 percent of the federal poverty level, 6.4 percent had incomes between 251 and 400, and 
1.9 percent had incomes above 400 percent of the federal poverty level. In the Graduated 
Verification comparison sites, 73.0 percent of students not eligible for free meals in our sample 
had incomes between 131 and 185 percent of poverty, 19.5 percent had incomes between 186 
and 250 percent of poverty, 6.6 percent had incomes between 251 percent and 400 percent of 
poverty, and 0.9 percent had incomes above 400 percent of poverty. In each set of comparison 
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TABLE IV. 16 



ESTIMATED IMPACT OE GRADUATED VERIEICATION PILOT PROJECTS ON TARGETING EEEICIENCY 

(Standard Errors in Parentheses) 



Regression-Adjusted Percentage of All 
Students 



Measure of Targeting Efficiency 



Pilot Comparison Impact 



Targeting Efficiency Measures Among Non- 80.0 

Directly Certified Students (CTE_3a) 



83.2 -3.2 

(2.7) 



Notes: Definitions of measures are provided in Table II.2. Standard errors of the impact estimates were estimated 
using bootstrapping methods, whereby 1,000 subsamples were selected with replacement from the relevant 
analysis sample, the impact estimates were computed with each subsample, and the variance of the estimates 
in these repeated calculations was computed. 

Although we were unable to estimate impacts on the first targeting efficiency among non-directly certified 
students (CTE_la) for this report, we plan to estimate these impacts and include them in a supplemental 
appendix to this report to be completed later. 



*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 
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INCOME DISTRIBUTION OF CERTIFIED INELIGIBLE STUDENTS IN PILOT AND COMPARISON SITES 
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Note; Definitions of certification measures are provided in Table II.2. Estimates are based on weighted tabulations in which differing sample selection 
probabilities of the different meal-price status groups are used. 




sites, well over half the sample who were certified for free meals but ineligible for them had 
incomes in a range that made the student eligible for reduced-price meals. 

D. SUMMARY 

In this section, we briefly summarize the impact findings for the Up-Front Documentation 
and Graduated Verification pilot projects. 

1. Up-Front Documentation 

In the Up-Front Documentation pilots, the estimated impacts of the demonstration on 
deterrence were small in absolute terms or close to zero, and not statistically significant. Among 
households with incomes greater than 130 percent of poverty, the estimated impact of the pilot 
intervention on the percentage certified for free meals was -0.8 percentage points. Among those 
with incomes greater than 185 percent of poverty the estimated impact was essentially zero. We 
note that although -0.8 percentage point is small in absolute terms, if this were correct, it would 
represent a substantial proportionate reduction in the percentage of students receiving free meal 
benefits who were not eligible for them on the order of 20 percent. However, the estimate is 
imprecise and could be due to sampling error. Estimated impacts on families eligible for free or 
reduced-price meals indicate that the Up-Front Documentation pilot created barriers for some 
eligible families. We defined eligible families in three different ways, and found statistically 
significant barrier effects for two of the measures. The third measure also produced an estimate 
suggesting barriers, but was not statistically significant at the .05 level (Table IV. 16). For 
households receiving TANF or food stamps but not directly certified, the estimated impact was 
also negative but it was not statistically significant. 

The estimated impacts on accuracy and targeting were in the range of two to three 
percentage points, some estimates were positive and some were negative depending on the 
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precise measure, and none was statistically significant. Overall, we conclude that Up-Front 
Documentation has no appreciable effect on accuracy or targeting. 

2. Graduated Verification 

The Graduated Verification pilot projects exhibit a similar pattern of findings to those of the 
Up-Front Documentation pilot projects. Point estimates of the impacts on deterrence are mixed. 
For households above 130 percent of poverty, the estimated impact is essentially zero. For 
households above 185 percent of poverty, the estimated impact is -2.5 percentage points (or 
nearly 30 percent of the comparison group mean), but not statistically significant. We can be 
confident that impacts of 5 percentage points or larger did not occur, but because of limited 
sample sizes, we cannot be confident that smaller ones did not occur. 

Two measures of certification rates among students from families with incomes less than 
130 percent of poverty indicate that Graduated Verification created barriers for this group. A 
measure of certification among students from families with incomes less than 185 percent of 
poverty also indicates that Graduated Verification reduced certification for this group, although 
the impact estimate was not statistically significant at the .05 level. The estimated impacts on 
certification among students receiving TANF or food stamps also shows that certification rates 
of this group declined, but the estimated impacts are not statistically significant. 

Finally, the impacts on accuracy and targeting were in the range of three to four percentage 
points. Some were positive, some were negative, and none was statistically significant. 
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